From 4b1c85a5665e17d5883e544e1accdc0d913d0f2d Mon Sep 17 00:00:00 2001 From: n30nex Date: Sun, 20 Sep 2026 00:05:44 -0400 Subject: [PATCH 01/69] docs(workflow): share parity roadmap and tested review-stack tooling --- .github/workflows/contributor-tools.yml | 21 ++ .gitignore | 2 + CONTRIBUTOR_WORKFLOW.md | 92 ++++++ README.md | 1 + ROADMAP.md | 129 ++++++++ tools/beacon_stack.py | 393 ++++++++++++++++++++++++ tools/review-stack-web.example.json | 10 + tools/review-stack.example.json | 7 + tools/test_beacon_stack.py | 171 +++++++++++ 9 files changed, 826 insertions(+) create mode 100644 .github/workflows/contributor-tools.yml create mode 100644 .gitignore create mode 100644 CONTRIBUTOR_WORKFLOW.md create mode 100644 ROADMAP.md create mode 100644 tools/beacon_stack.py create mode 100644 tools/review-stack-web.example.json create mode 100644 tools/review-stack.example.json create mode 100644 tools/test_beacon_stack.py diff --git a/.github/workflows/contributor-tools.yml b/.github/workflows/contributor-tools.yml new file mode 100644 index 0000000..73f8a1e --- /dev/null +++ b/.github/workflows/contributor-tools.yml @@ -0,0 +1,21 @@ +name: Contributor workflow checks + +on: + pull_request: + paths: ["tools/**", ".github/workflows/contributor-tools.yml"] + push: + branches: [main] + paths: ["tools/**", ".github/workflows/contributor-tools.yml"] + +permissions: + contents: read + +jobs: + test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + - run: python -m unittest discover -s tools -p test_beacon_stack.py -v diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..7a60b85 --- /dev/null +++ b/.gitignore @@ -0,0 +1,2 @@ +__pycache__/ +*.pyc diff --git a/CONTRIBUTOR_WORKFLOW.md b/CONTRIBUTOR_WORKFLOW.md new file mode 100644 index 0000000..7f38c90 --- /dev/null +++ b/CONTRIBUTOR_WORKFLOW.md @@ -0,0 +1,92 @@ +# Contributor workflow + +Use this workflow to keep small Beacon changes reviewable while reducing manual branch maintenance. The executable helper is [tools/beacon_stack.py](tools/beacon_stack.py); its [offline regression tests](tools/test_beacon_stack.py) run in this repository's CI. + +## Working loop + +1. Refresh issues, review feedback and the [roadmap](ROADMAP.md). Choose one logical API, page or correction. +2. Start an isolated feature worktree. After the old queue has merged, Start uses freshly fetched `dev` and performs no rebase or build. If a pending stack exists, Start uses its published tip and reports the dependency; use Check to verify that queue's current validation state. +3. Follow each repository's contribution rules. Prefer existing components and feature-owned files; keep shared startup/router/navigation changes in a declared order. +4. Build and test the change, open its PR against `dev`, then record its exact parent, head, fork branch and worktree in the manifest. Link the parent-to-head comparison in the PR body. +5. Keep current Beacon contribution PRs out of draft as requested by the contributor, with dependencies visible. Request MrAlders0n's review; if account permissions prevent formal assignment, use an explicit review-request comment instead. +6. After merges, run Sync and Check. Rebuild/redeploy the preview only if the composed source changes, preserving the actual running revision and corresponding-source offer. + +A source conflict still needs review. The helper automates routine history movement; it does not promise that overlapping edits can never conflict, merge upstream PRs, deploy services, or create scheduled jobs. + +## Setup + +Requires Python 3.10+, Git and an authenticated GitHub CLI. Server validation needs Go and Swag; web validation needs Node/npm. Use the repository's pinned dependency/toolchain requirements. + +Clone the application repositories with `origin` pointing at MeshCore-Beacon and a contribution remote pointing at your fork. The helper checks both identities before work. Keep local manifests, logs and isolated worktrees in a workspace outside the docs checkout, for example: + +```text +workspace/ + beacon-server/ + beacon-web/ + planning/ + review-stack.json + review-stack-web.json + evidence/ + worktrees/ +``` + +Copy [the server template](tools/review-stack.example.json) and [web template](tools/review-stack-web.example.json) into `planning/`, replacing the account/fork names. Paths may be absolute or relative to `--workspace`. Start with an empty `entries` list when no contribution queue exists. + +```bash +python tools/beacon_stack.py status --workspace /path/to/workspace +python tools/beacon_stack.py start --workspace /path/to/workspace --branch codex/next-change +python tools/beacon_stack.py sync --workspace /path/to/workspace +python tools/beacon_stack.py check --workspace /path/to/workspace +``` + +Add `--project web` for the web queue. `--manifest` and `--state` select explicit files/directories when needed. Start writes its exact parent/worktree to `evidence/review-stack/started.json` (or the web equivalent); it does not automatically create a PR or invent a manifest entry. + +An entry records the feature's delta boundary, not a guessed merge base: + +```json +{ + "pr": 123, + "branch": "codex/my-feature", + "base": "EXACT_PARENT_COMMIT", + "head": "CURRENT_LOCAL_COMMIT", + "remote_head": "CURRENT_PUBLISHED_COMMIT", + "worktree": "worktrees/my-feature" +} +``` + +`remote_head` is the lease protecting someone else's newer work. Never overwrite it to suppress a mismatch. Review external changes before updating the manifest. + +## What Sync actually does + +- Reads current `dev` and PR state, drops merged parents and rejects unexpectedly changed or unmerged-closed PRs. +- Replays only each pending feature's declared delta in an isolated worktree. Existing dirty work is preserved. +- Regenerates generated-only server Swagger conflicts. Authored source conflicts stop with the worktree intact; it never blindly chooses a source-code side. +- Reuses local validation only for the same entire Git tree, tool versions, platform and effective build settings. Changed inputs run the repository checks. Real PostgreSQL, browser and native/device evidence remain separate requirements. +- Rechecks upstream and PR state immediately before publishing, including after a long validation run. If a merge happened meanwhile, it stops and retains reusable receipts. +- Publishes changed fork branches atomically with explicit leases, verifies remote heads and reports current CI separately. + +`refresh` prepares without pushing; `publish` publishes a prepared result after fresh checks. `sync` combines both. `check` requires current published heads and passing required checks; only explicitly configured skipped jobs are allowed. + +Local validation reuse does not suppress GitHub's checks on a changed head. The first refresh after an independent change joins several candidates can require new combination checks; later identical-tree refreshes reuse those receipts. + +## Preview records and independent work + +Set `preview.server_tree` to the verified composed Git tree only after native/public validation. This field is also used in the web manifest for compatibility. A different commit with an identical tree does not require rebuilding or relabeling an existing artifact. Keep the real built revision/source archive. + +Independent pending work can be listed in `preview_overlays` as `{"pr": 123, "head": "EXACT_COMMIT"}`. Merged overlays drop automatically; changed or unmerged-closed overlays stop composition. Keep overlapping changes in the main ordered queue. + +After all queued changes merge, Sync leaves an empty queue. Start then creates the next branch directly from current `dev`. This is the normal path for a new phase, without carrying historical feature commits forward. + +## Optional site guard + +The helper always requires clean, isolated feature branches, matching remotes and unchanged publication inputs. Operators with an additional repository guard can set `guard_script` in the local manifest, supply `--guard-script /path/to/guard.ps1`, or set `BEACON_GUARD_SCRIPT`. A manifest guard takes precedence; a configured missing or failing guard stops work. The hook receives Preflight/Finish, repository and receipt arguments through PowerShell. + +The Canadaverse workspace keeps its required site guard in the local wrapper. Public examples contain no private host, key, credential or workstation path. + +## Validate the helper + +```bash +python -m unittest discover -s tools -p test_beacon_stack.py -v +``` + +Tests cover squash/drop-parent behavior, a fresh phase after all merges, cache reuse, environment changes, independent overlays, dirty/default-branch rejection, source conflicts and upstream movement during validation. They use disposable local Git repositories and no GitHub or Pi credentials. diff --git a/README.md b/README.md index 1c12f3a..6ab0285 100644 --- a/README.md +++ b/README.md @@ -6,6 +6,7 @@ This repo is the single place to: 1. **Grab a deployment** — copy the Docker Compose folder for the topology you want, fill in your variables, and `docker compose up -d`. 2. **Read the docs** — project-wide design and API documentation that describe how the whole system works. +3. **Follow development** — the [parity and analytics roadmap](ROADMAP.md) and [executable contributor workflow](CONTRIBUTOR_WORKFLOW.md) track review dependencies, validation and the next phases. --- diff --git a/ROADMAP.md b/ROADMAP.md new file mode 100644 index 0000000..2fa43df --- /dev/null +++ b/ROADMAP.md @@ -0,0 +1,129 @@ +# Beacon parity and analytics roadmap + +Updated 20 September 2026. This is the working roadmap for n30nex's ongoing contributions toward CoreScope feature parity. Maintainers decide acceptance and merge order; deployment owners handle the production switch. + +Refresh GitHub issues, PR feedback and branch state before starting a phase. This document is a snapshot, and linked issues/PRs are the current source of truth. + +## Direction + +Bring useful CoreScope investigation and analytics features into Beacon's existing ingest, database, API, cache and web components. Prioritize review regressions and measured stability/performance problems, then useful analytics pages. Keep each API or page a focused contribution with explicit counting semantics and validation. + +Current sites: + +- [Beacon reference deployment](https://dev.meshcore.ca/) +- [CoreScope reference deployment](https://live.meshcore.ca/) +- [Development preview](https://canadaverse.org/beacon-dev/) and its [changelog/source](https://canadaverse.org/beacon-dev/source.html) + +| Repository | Responsibility | Contribution target | +|---|---|---| +| [beacon-server](https://github.com/MeshCore-Beacon/beacon-server) | Ingest, storage, read models and public/protected APIs | `dev` | +| [beacon-web](https://github.com/MeshCore-Beacon/beacon-web) | Investigation tools and analytics pages | `dev` | +| [beacon-docs](https://github.com/MeshCore-Beacon/beacon-docs) | Shared contracts, operator guidance and this roadmap | `main` | +| [beacon-mobile](https://github.com/MeshCore-Beacon/beacon-mobile) | Mobile client; coordinate API compatibility | `main` | + +## Current review queue + +All active contribution PRs are available for review without draft status at the contributor's request. Dependencies still determine merge order. Review requests do not establish approval, and required checks must pass on the current published head. + +Server order: **#149 → #154 → #157 → #159**. + +| PR | Scope | Issue disposition | +|---|---|---| +| [Server #149](https://github.com/MeshCore-Beacon/beacon-server/pull/149) | Protected operator-account lifecycle | Partial #60; these records are not browser logins | +| [Server #154](https://github.com/MeshCore-Beacon/beacon-server/pull/154) | Protected database/config backup download | Partial #72; login, validation/import and deployment-file coverage remain | +| [Server #157](https://github.com/MeshCore-Beacon/beacon-server/pull/157) | Bounded SNR/RSSI distributions and hourly statistics | Complete endpoint scope in #156 | +| [Server #159](https://github.com/MeshCore-Beacon/beacon-server/pull/159) | Bounded received-path and hash-width statistics | Complete endpoint scope in #158 | + +Web order: **#52 → #53 → #55 → #57**. Observer comparison [#48](https://github.com/MeshCore-Beacon/beacon-web/pull/48) and foreign-node display [#49](https://github.com/MeshCore-Beacon/beacon-web/pull/49) are merged; issue #26 is closed. + +| PR | Scope | Dependency / issue | +|---|---|---| +| [Web #52](https://github.com/MeshCore-Beacon/beacon-web/pull/52) | Traffic heatmap, hourly trends and reception share | #50 | +| [Web #53](https://github.com/MeshCore-Beacon/beacon-web/pull/53) | Regional scope charts and exact counts | After #52; #51 | +| [Web #55](https://github.com/MeshCore-Beacon/beacon-web/pull/55) | RF / Signal charts and sample availability | After #53 and server #157; #54 | +| [Web #57](https://github.com/MeshCore-Beacon/beacon-web/pull/57) | Paths & Hashes charts and classification coverage | After #55 and server #159; #56 | + +The router foundation [server #155](https://github.com/MeshCore-Beacon/beacon-server/pull/155) is merged. The refresh workflow drops accepted squash parents, preserves focused feature deltas and handles changed branch history in isolated worktrees. See [the executable contributor workflow](CONTRIBUTOR_WORKFLOW.md). + +## Delivered foundations + +- Faster bounded node, route, trace and clock-stat queries; list-limit validation and NULL-observation handling. Representative changes: [#111](https://github.com/MeshCore-Beacon/beacon-server/pull/111), [#118](https://github.com/MeshCore-Beacon/beacon-server/pull/118), [#120](https://github.com/MeshCore-Beacon/beacon-server/pull/120), [#122](https://github.com/MeshCore-Beacon/beacon-server/pull/122), [#124](https://github.com/MeshCore-Beacon/beacon-server/pull/124). +- MQTT client isolation, proxy identity handling, API/WebSocket limits and interrupted-index recovery. Timeout attribution in #116 remains separate from these accepted fixes. +- Observer age-out, advert summaries, endpoint snapshots and companion matching, stable channel paging, packet search and shared packet links. +- Observer activity/telemetry and comparison, regional scope statistics, runtime administration, optional foreign-repeater detection and the backup-export foundation. + +## Analytics delivery and counting rules + +| Page | Current behavior | Important interpretation | +|---|---|---| +| [Traffic](https://canadaverse.org/beacon-dev/?tab=Analytics&statsTab=traffic&range=24h) | UTC hourly heatmap, IATA trends, reception share and exact counts | Counts reported receptions; missing hourly records remain gaps | +| [Scopes](https://canadaverse.org/beacon-dev/?tab=Analytics&statsTab=scopes) | Regional packet, observer-membership and default-scope-node charts | Retained counts have no rolling date filter; memberships can overlap | +| [RF / Signal](https://canadaverse.org/beacon-dev/?tab=Analytics&statsTab=signal&range=24h) | SNR/RSSI distributions, hourly means, sample coverage and exact tables | Missing/non-finite and unavailable zero/zero readings are excluded per metric; a measured zero SNR remains valid | +| [Paths & Hashes](https://canadaverse.org/beacon-dev/?tab=Analytics&statsTab=paths&range=24h) | Hash-width share, received header-entry distribution, hourly trends and coverage | Empty paths never vote for width; TRACE paths hold signal readings; unusable metadata is unclassified | + +The path page counts stored receptions, not unique devices. Flood paths accumulate entries, while direct routes carry remaining entries. Observed widths do not establish device capability or collision rates. Signal readings describe reception at the reporting observer rather than a complete end-to-end link. + +Both new aggregate APIs accept explicit windows of at most 30 days, propagate cancellation, use region-specific caching and have a 15-second execution bound. A raw 30-day global query can still take seconds at larger volumes. On a shared Pi with one million synthetic rows, signal queries measured about 2.1–6.5 seconds and path queries 2.8–7.5 seconds across prepared-plan modes. Measure production volume before choosing precomputed distributions or claiming release readiness. + +The current combined preview has passed native Go/PostgreSQL/HTTP validation and **781 web tests**, plus desktop/mobile layout checks at 320/390/768/1280px. Synthetic fixtures tested gaps, trace/unclassified data, zero samples, errors/retry and long windows; they were not published. Current revisions and corresponding-source archives are on the preview's changelog page. + +## Next phases + +1. **Review feedback and regressions first.** Refresh both application queues, resolve attributable failures, and retain explicit issue scope. +2. **Channel Activity analytics.** Inspect existing aggregates and add bounded channel traffic/trend views. Define messages versus receptions, unknown/encrypted-channel coverage and the behavior when keys are unavailable. Keep keys and message contents out of aggregate responses; avoid raw-message paging to build statistics. +3. **Observed ambiguity and topology.** Separate static prefix conflicts from observed unresolved/ambiguous paths. Add justified route-pattern, neighbour, hop and distance views with clear provenance and bounded work. +4. **Queued administration and backup slices.** Add non-destructive archive validation, then resolve browser login/session, import and deployment-file coverage separately. Test restoration against disposable databases. +5. **Production evidence and handoff.** Reconcile history, operational limits and the parity matrix below before preparing a release/cutover handoff. + +## Listed work still open + +| Issue | Remaining scope | +|---|---| +| [Server #116](https://github.com/MeshCore-Beacon/beacon-server/issues/116) | Attribute repeated MQTT ping timeouts to a measured cause; fresh traffic alone is insufficient | +| [Server #99](https://github.com/MeshCore-Beacon/beacon-server/issues/99) | Define and complete remaining packet-type summary coverage after advert summaries | +| [Server #60](https://github.com/MeshCore-Beacon/beacon-server/issues/60) | Remaining administration/worker/persistence behavior; account records do not establish login sessions | +| [Server #72](https://github.com/MeshCore-Beacon/beacon-server/issues/72) | Archive validation/import, browser access, deployment-file coverage and remote/scheduled backup scope | +| [Web #12](https://github.com/MeshCore-Beacon/beacon-web/issues/12) | Supported-language and formatting scope for internationalization | + +The analytics endpoint/page issues remain open while their corresponding PRs await merge. Use closing references only when a PR completes the issue's accepted scope; use related references for partial work. + +## Production parity matrix + +Matching tab names is not acceptance. Each capability needs verified semantics, time/region behavior, empty/partial data, performance and browser evidence. + +| Capability | Coverage / remaining evidence | +|---|---| +| Overview | Mesh overview and Traffic; reconcile packet/reception grain and retained windows | +| RF / Signal | New distributions, weighted means and sample coverage; production-volume measurements remain | +| Topology / route patterns | Existing routes, traces and neighbour graph; deeper edge/subpath/connectivity analysis remains | +| Channels | Directory/chat/talkers exist; traffic statistics, unknown-channel and history behavior remain | +| Hash statistics | Paths & Hashes covers observed ordinary widths and entries; trace-payload widths are distinct | +| Hash issues | Endpoint/path ambiguity primitives exist; observed ambiguity and static conflicts need separate views | +| Node analytics | Existing directory/detail/observations; richer attributed activity, signal, payload and peer analysis remains | +| My Repeaters | Owner selection/watchlist behavior and grouped analytics need an agreed contract | +| Repeater metrics | Observer telemetry overlaps partially; units, resets and role attribution need reconciliation | +| Distance | Coordinates/maps exist; valid link/path distances, unknown positions and confidence remain | +| Neighbour graph | Existing graph needs retained scaling, filter and accessible-fallback acceptance evidence | +| RF health | Noise/airtime/error telemetry exists; comparable health views need real samples and valid deltas | +| Clock health | Existing clock-drift endpoint/UI; retain role, threshold and history semantics | +| Roles | Node-type census exists; activity and unknown-role interpretation need acceptance evidence | +| Scopes | Regional API and new page exist; reconcile against populated retained data | +| Prefix tool | Public-prefix inspection/simulation remains unverified | +| Observer comparison | Merged backend/web; retain distinct-packet, time/region and zero-data tests | +| Other workflows | Validate decoder/search/sharing, live map/replay, settings, optional clients and legacy links | + +## Release gates + +- Inventory actual CoreScope retention, earliest/latest durable data, configuration and recovery copies. Example retention settings and public in-memory counts are not production history evidence. +- Reconcile Beacon's deduplication, identity, encryption/key and time semantics. Document whether migration or sufficient parallel ingestion supplies each historical window. +- Complete backup recovery scope with private disposable restore tests, including schema/data/sequence continuation and the deliberately excluded deployment files. +- Measure cold start, reconnects, ingest freshness, CPU/memory/storage, query latency and maintenance against representative production volume. Long-window raw aggregates may need rollups. +- Validate keyboard/mobile/browser behavior, chart readability and sharing. Physical iPhone Safari and a sustained load/connection soak remain open validation gaps. +- Verify release artifacts from reviewed source and applicable CI. The upstream web CodeQL workflow is currently disabled; its skipped job does not count as a security scan. +- Publish matching source, configuration guidance, known limitations and a verified rollback procedure. The deployment owner performs the production switch. + +The development preview currently has short retained history and no populated transport-scope records. Its public admin/backup and foreign detection are disabled. These limitations remain explicit until configuration and validation support enabling them. + +## Keeping this roadmap useful + +Update this file when a phase is delivered, a dependency merges, an issue closes or the next priority changes. Keep private configuration and host-specific operational records outside this repository. Link current GitHub work and the public source/changelog so another contributor can continue without a private workstation path. diff --git a/tools/beacon_stack.py b/tools/beacon_stack.py new file mode 100644 index 0000000..dcd749d --- /dev/null +++ b/tools/beacon_stack.py @@ -0,0 +1,393 @@ +"""Refresh the declared Beacon PR stack without touching unrelated work.""" +import argparse +import hashlib +import json +import os +import re +from pathlib import Path +import subprocess +import shutil +import sys +import uuid + +ROOT = Path(__file__).resolve().parents[1] +MANIFEST = ROOT / 'planning/review-stack.json' +STATE = ROOT / 'evidence/review-stack' +GENERATED = {'docs/docs.go', 'docs/swagger.json', 'docs/swagger.yaml'} +ENV = dict(os.environ, GOMAXPROCS='2', GOFLAGS='-p=2', GIT_EDITOR='true') +# Cached local checks are unit checks. Database verification uses the explicit +# private Pi/CI workflows, never an accidentally inherited connection target. +ENV.pop('BEACON_TEST_POSTGRES_DSN', None) +ENV['BEACON_BACKUP_TEST_POSTGRES'] = '0' +GUARD_SCRIPT = os.environ.get('BEACON_GUARD_SCRIPT') + + +def local_path(value): + path = Path(value) + return (path if path.is_absolute() else ROOT/path).resolve() + + +def check_remotes(repo, manifest): + if any(not re.fullmatch(r'[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+', manifest[key]) for key in ('upstream', 'fork')): + raise RuntimeError('Expected GitHub owner/repository names') + def slug(remote): + url = git(repo, 'remote', 'get-url', remote).stdout.strip().removesuffix('.git') + for prefix in ('https://github.com/', 'git@github.com:'): + if url.startswith(prefix): + return url[len(prefix):].lower() + return None + if slug('origin') != manifest['upstream'].lower() or slug(manifest['remote']) != manifest['fork'].lower(): + raise RuntimeError('Repository remotes do not match the declared upstream and contribution fork') + + +def command(args, cwd=None, check=True, log=None): + result = subprocess.run(args, cwd=cwd, env=ENV, text=True, encoding='utf-8', + stdout=subprocess.PIPE, stderr=subprocess.STDOUT) + if log: + Path(log).write_text(result.stdout, encoding='utf-8') + if check and result.returncode: + raise RuntimeError(f"{args[0]} {args[1]} failed in {cwd or ROOT}:\n{result.stdout}") + return result + + +def git(repo, *args, check=True): + return command(['git', *args], cwd=repo, check=check) + + +def write(path, value): + path.parent.mkdir(parents=True, exist_ok=True) + temporary = path.with_suffix(path.suffix + '.new') + temporary.write_text(json.dumps(value, indent=2) + '\n', encoding='utf-8') + temporary.replace(path) + + +def active_entries(entries, pulls): + active = [] + for entry, pull in zip(entries, pulls, strict=True): + if pull['merged']: + continue + if pull['state'] != 'open': + raise RuntimeError(f"PR #{entry['pr']} was closed without merging; resolve its disposition first") + if pull['head']['sha'] != entry['remote_head'] or pull['head']['ref'] != entry['branch']: + raise RuntimeError(f"PR #{entry['pr']} changed outside this stack; inspect before updating") + if pull['base']['ref'] != 'dev': + raise RuntimeError('Unexpected PR target; this workflow only targets dev') + if any(not re.fullmatch(r'[a-f0-9]{40}', entry.get(key, '')) for key in ('base', 'head', 'remote_head')): + raise RuntimeError('Stack boundaries must be exact commit IDs') + if entry['branch'] in ('main', 'master', 'dev'): + raise RuntimeError('Contribution branches must not be default or release branches') + active.append(entry.copy()) + return active + + +def guard(path, mode, receipt): + path = local_path(path) + branch = git(path, 'branch', '--show-current').stdout.strip() + if not branch or branch in ('main', 'master', 'dev') or git(path, 'status', '--porcelain').stdout: + raise RuntimeError(f'Clean, isolated feature branch required: {path}') + if GUARD_SCRIPT: + if not Path(GUARD_SCRIPT).is_file(): + raise RuntimeError('Configured repository guard is missing; no changes are permitted') + command(['pwsh', '-NoProfile', '-File', GUARD_SCRIPT, '-Mode', mode, '-RepoPath', str(path), + '-Base', 'origin/dev', '-NoRemote', '-OutputPath', str(receipt)], log=receipt.with_suffix('.log')) + if not json.loads(receipt.read_text(encoding='utf-8-sig'))['ready']: + raise RuntimeError('Repository guard did not pass') + else: + write(receipt, dict(ready=True, branch=branch, head=git(path, 'rev-parse', 'HEAD').stdout.strip(), clean=True)) + + +def verify_publish_state(manifest, plan): + # Validation can take minutes. Refresh these inputs immediately before the + # leased push, including Sync mode, rather than trusting its initial snapshot. + base = command(['gh', 'api', f"repos/{manifest['upstream']}/commits/dev", '--jq', '.sha']).stdout.strip() + pulls = [json.loads(command(['gh', 'api', f"repos/{manifest['upstream']}/pulls/{e['pr']}"]).stdout) for e in manifest['entries']] + if any(p['head']['repo']['full_name'] != manifest['fork'] for p in pulls): + raise RuntimeError('The contribution fork changed during validation') + active = active_entries(manifest['entries'], pulls) + if plan['base'] != base or [e['pr'] for e in plan['entries']] != [e['pr'] for e in active]: + raise RuntimeError('Upstream or merge state changed during validation; refresh again. Verified trees remain cached.') + + +def rebase(path, parent, previous_base, kind='server'): + result = git(path, 'rebase', '--onto', parent, previous_base, check=False) + while result.returncode: + conflicts = set(git(path, 'diff', '--name-only', '--diff-filter=U').stdout.splitlines()) + if kind != 'server' or not conflicts or not conflicts <= GENERATED: + raise RuntimeError(f"Source conflict preserved in {path}: {', '.join(sorted(conflicts)) or result.stdout}. No branches were published.") + # Generated contracts are rebuilt from the already merged source. Never + # choose a side automatically for application code or authored docs. + git(path, 'restore', '--ours', '--worktree', '--', *sorted(conflicts)) + command(['swag', 'init', '-g', 'cmd/beacon/main.go', '-o', 'docs', '--parseInternal', '--parseDependency'], cwd=path) + git(path, 'add', '--', *sorted(GENERATED)) + result = git(path, '-c', 'core.editor=true', 'rebase', '--continue', check=False) + + +def validate(path, cache): + if git(path, 'status', '--porcelain').stdout: + raise RuntimeError(f'Dirty worktree preserved: {path}') + versions = command(['go', 'version'], cwd=path).stdout.strip() + ' / ' + command(['swag', '--version'], cwd=path).stdout.strip() + build_environment = command(['go', 'env', '-json', 'GOOS', 'GOARCH', 'CGO_ENABLED', 'GOFLAGS', 'GOWORK'], cwd=path).stdout.strip() + tree = git(path, 'rev-parse', 'HEAD^{tree}').stdout.strip() + receipt = cache / (tree + '.json') + if receipt.exists(): + record = json.loads(receipt.read_text(encoding='utf-8')) + if record['tools'] == versions and record.get('build_environment') == build_environment and record['passed']: + return dict(tree=tree, reused=True, receipt=str(receipt)) + logdir = cache / tree + logdir.mkdir(parents=True, exist_ok=True) + command(['swag', 'init', '-g', 'cmd/beacon/main.go', '-o', 'docs', '--parseInternal', '--parseDependency'], cwd=path, log=logdir/'swagger.txt') + changed = set(git(path, 'diff', '--name-only').stdout.splitlines()) + if changed: + if not changed <= GENERATED: + raise RuntimeError('Unexpected non-generated changes during validation') + git(path, 'add', '--', *sorted(GENERATED)) + git(path, 'commit', '-m', 'docs(api): refresh generated contract after stack update') + tree = git(path, 'rev-parse', 'HEAD^{tree}').stdout.strip() + receipt = cache / (tree + '.json') + git(path, 'diff', '--check', 'origin/dev...HEAD') + if command(['gofmt', '-l', '.'], cwd=path).stdout.strip(): + raise RuntimeError('Unformatted Go files; correct them before publishing') + for name, args in [('build', ['go', 'build', './...']), ('vet', ['go', 'vet', './...']), ('tests', ['go', 'test', './...'])]: + command(args, cwd=path, log=logdir/(name+'.txt')) + write(receipt, dict(passed=True, tree=tree, tools=versions, build_environment=build_environment, logs=str(logdir))) + return dict(tree=tree, reused=False, receipt=str(receipt)) + + +def web_fingerprint(path): + settings = {key: value for key, value in ENV.items() if key.startswith('VITE_') or key in ('NODE_ENV', 'NODE_OPTIONS')} + for name in ('.env', '.env.local', '.env.production', '.env.production.local'): + if (path/name).is_file(): + settings[name] = hashlib.sha256((path/name).read_bytes()).hexdigest() + return hashlib.sha256(json.dumps(settings, sort_keys=True).encode()).hexdigest() + + +def validate_web(path, cache): + if git(path, 'status', '--porcelain').stdout: + raise RuntimeError(f'Dirty worktree preserved: {path}') + npm = shutil.which('npm.cmd' if os.name == 'nt' else 'npm') + if not npm: + raise RuntimeError('npm is unavailable') + versions = command(['node', '--version'], cwd=path).stdout.strip() + ' / ' + command([npm, '--version'], cwd=path).stdout.strip() + ' / ' + command(['node', '-p', 'process.platform+"/"+process.arch'], cwd=path).stdout.strip() + tree = git(path, 'rev-parse', 'HEAD^{tree}').stdout.strip() + fingerprint = web_fingerprint(path) + receipt = cache/(tree+'.json') + if receipt.exists(): + record = json.loads(receipt.read_text(encoding='utf-8')) + if record.get('tools') == versions and record.get('build_environment') == fingerprint and record['passed']: + return dict(tree=tree, reused=True, receipt=str(receipt)) + logs = cache/tree; logs.mkdir(parents=True, exist_ok=True) + modules = path/'node_modules' + if not modules.resolve().is_relative_to(path.resolve()): + raise RuntimeError('Refusing to replace dependencies linked outside this worktree') + lock = hashlib.sha256((path/'package-lock.json').read_bytes()).hexdigest() + marker = modules/'.beacon-lock-sha256' + if not marker.exists() or marker.read_text() != lock: + command([npm, 'ci', '--no-audit', '--no-fund'], cwd=path, log=logs/'install.txt') + marker.write_text(lock) + git(path, 'diff', '--check', 'origin/dev...HEAD') + for name, arguments in [('build', ['run', 'build']), ('lint', ['run', 'lint']), ('tests', ['test', '--', '--maxWorkers=2'])]: + command([npm, *arguments], cwd=path, log=logs/(name+'.txt')) + write(receipt, dict(passed=True, tree=tree, tools=versions, build_environment=fingerprint, logs=str(logs))) + return dict(tree=tree, reused=False, receipt=str(receipt)) + + +def preview_tree(repo, head, overlays): + for overlay in overlays: + tree = git(repo, 'merge-tree', '--write-tree', head, overlay).stdout.strip() + head = git(repo, 'commit-tree', tree, '-p', head, '-p', overlay, '-m', 'test: assemble preview source').stdout.strip() + return git(repo, 'rev-parse', head+'^{tree}').stdout.strip() + + +def active_overlays(manifest): + heads = [] + for overlay in manifest.get('preview_overlays', []): + if isinstance(overlay, str): + heads.append(overlay) + continue + pull = json.loads(command(['gh', 'api', f"repos/{manifest['upstream']}/pulls/{overlay['pr']}"]).stdout) + if pull['merged']: + continue + if pull['state'] != 'open' or pull['head']['sha'] != overlay['head'] or pull['head']['repo']['full_name'] != manifest['fork']: + raise RuntimeError('An independent preview candidate changed; review it before composing the preview') + heads.append(overlay['head']) + return heads + + +def simulate_squash_order(repo, base, entries): + """Prove the history-only refresh after each earlier squash preserves code.""" + parent = base + proof = [] + for entry in entries: + parent_tree = git(repo, 'rev-parse', parent+'^{tree}').stdout.strip() + delta_base_tree = git(repo, 'rev-parse', entry['base']+'^{tree}').stdout.strip() + if parent_tree != delta_base_tree: + raise RuntimeError('Stack parent differs from the reviewed delta base') + tree = git(repo, 'rev-parse', entry['head']+'^{tree}').stdout.strip() + # Squashing changes ancestry. A child must drop its accepted parent, + # even when the source is identical. Here identical parent trees prove + # this is a topology-only move; refresh performs the real rebase --onto. + restacked = git(repo, 'commit-tree', tree, '-p', parent, '-m', 'test: simulate history-only stack refresh').stdout.strip() + merged = git(repo, 'merge-tree', '--write-tree', parent, restacked).stdout.strip() + if merged != tree: + raise RuntimeError('Simulated merge did not produce the reviewed candidate tree') + parent = git(repo, 'commit-tree', tree, '-p', parent, '-m', 'test: simulate reviewed stack squash merge').stdout.strip() + proof.append(dict(pr=entry['pr'], tree=tree, clean_after_squash_and_refresh=True, source_tree_preserved=True)) + return proof + + +def main(mode, manifest_path=MANIFEST, state=STATE, branch=None): + global GUARD_SCRIPT + manifest = json.loads(manifest_path.read_text(encoding='utf-8')) + if manifest.get('guard_script'): + GUARD_SCRIPT = str(local_path(manifest['guard_script'])) + if GUARD_SCRIPT and not Path(GUARD_SCRIPT).is_file(): + raise RuntimeError('Configured repository guard is missing; no changes are permitted') + kind = manifest.get('kind', 'server') + repo = local_path(manifest['repo']) + check_remotes(repo, manifest) + pulls = [json.loads(command(['gh', 'api', f"repos/{manifest['upstream']}/pulls/{e['pr']}"]).stdout) for e in manifest['entries']] + for pull in pulls: + if pull['head']['repo']['full_name'] != manifest['fork']: + raise RuntimeError('Unexpected fork identity') + entries = active_entries(manifest['entries'], pulls) + base = command(['gh', 'api', f"repos/{manifest['upstream']}/commits/dev", '--jq', '.sha']).stdout.strip() + parent = base + needs_refresh = False + for entry in entries: + needs_refresh = needs_refresh or entry['base'] != parent + parent = entry['head'] + if mode == 'check': + if needs_refresh: + raise RuntimeError('The stack has an outdated parent; refresh before relying on previous checks') + results = [] + for entry in entries: + result = command(['gh', 'pr', 'checks', str(entry['pr']), '-R', manifest['upstream'], + '--json', 'name,state,bucket,link'], check=False) + checks = json.loads(result.stdout) if result.stdout.lstrip().startswith('[') else [] + passed = bool(checks) and all(c['bucket'] == 'pass' or (c['bucket'] == 'skipping' and c['name'] in manifest.get('allowed_skips', [])) for c in checks) and any(c['name'] == 'build' and c['bucket'] == 'pass' for c in checks) + results.append(dict(pr=entry['pr'], head=entry['remote_head'], + source_verified=entry['head'] == entry['remote_head'], passed=passed, checks=checks)) + write(state/'github-checks.json', results) + print(json.dumps(results, indent=2)) + if not all(r['source_verified'] and r['passed'] for r in results): + raise RuntimeError('The current stack still has unpublished changes or incomplete/failing GitHub checks') + return + if mode == 'status': + print(json.dumps(dict(upstream=base, order=[e['pr'] for e in entries], needs_refresh=needs_refresh, + merged=[e['pr'] for e, p in zip(manifest['entries'], pulls) if p['merged']], + unpublished=[e['pr'] for e in entries if e['head'] != e['remote_head']]), indent=2)) + return + state.mkdir(parents=True, exist_ok=True) + if mode == 'start': + if not branch: + raise RuntimeError('Start requires --branch with a new feature branch name') + if not re.fullmatch(r'[A-Za-z0-9][A-Za-z0-9._/-]*', branch): + raise RuntimeError('Use a plain feature branch name') + if needs_refresh: + raise RuntimeError('Refresh the existing stack before starting another dependent change') + if any(e['head'] != e['remote_head'] for e in entries): + raise RuntimeError('Publish the prepared stack before starting another dependent change') + git(repo, 'check-ref-format', '--branch', branch) + if branch in ('main', 'master', 'dev'): + raise RuntimeError('Choose a feature branch, not a default or release branch') + git(repo, 'fetch', 'origin', 'dev') + if git(repo, 'rev-parse', 'origin/dev').stdout.strip() != base: + raise RuntimeError('Upstream moved; start again from a fresh snapshot') + if entries: + git(repo, 'fetch', manifest['remote']) + parent = entries[-1]['head'] if entries else base + path = ROOT/'worktrees'/('feature-'+uuid.uuid4().hex[:10]) + if not path.resolve().is_relative_to(ROOT.resolve()): + raise RuntimeError('Worktree directory escapes the selected workspace') + git(repo, 'worktree', 'add', '-b', branch, str(path), parent) + guard(path, 'Preflight', state/'start-guard.json') + result = dict(branch=branch, parent=parent, worktree=str(path), depends_on=entries[-1]['pr'] if entries else None) + write(state/'started.json', result) + print(json.dumps(result, indent=2)) + return + if mode in ('refresh', 'sync'): + git(repo, 'fetch', 'origin', '--prune') + git(repo, 'fetch', manifest['remote']) + if git(repo, 'rev-parse', 'origin/dev').stdout.strip() != base: + raise RuntimeError('Upstream moved during refresh; run it again') + parent = base + for entry in entries: + path = local_path(entry['worktree']) + common = lambda checkout: (checkout/Path(git(checkout, 'rev-parse', '--git-common-dir').stdout.strip())).resolve() + if common(path) != common(repo): + raise RuntimeError('Recorded worktree belongs to a different repository') + if git(path, 'status', '--porcelain').stdout or git(path, 'rev-parse', 'HEAD').stdout.strip() != entry['head']: + raise RuntimeError(f"Unrecorded work preserved at {path}; record the intended commit before refreshing") + if parent != entry['base']: + suffix = uuid.uuid4().hex[:10] + path = ROOT/'worktrees'/f"stack-{entry['pr']}-{suffix}" + if not path.resolve().is_relative_to(ROOT.resolve()): + raise RuntimeError('Worktree directory escapes the selected workspace') + git(repo, 'worktree', 'add', '-b', f"codex/stack-{entry['pr']}-{suffix}", str(path), entry['head']) + guard(path, 'Preflight', state/f"preflight-{entry['pr']}.json") + rebase(path, parent, entry['base'], kind) + entry['worktree'] = str(path) + else: + guard(path, 'Preflight', state/f"preflight-{entry['pr']}.json") + entry['base'] = parent + entry['validation'] = (validate_web if kind == 'web' else validate)(path, state/'validation') + entry['head'] = git(path, 'rev-parse', 'HEAD').stdout.strip() + if entry['head'] == parent: + raise RuntimeError(f"Open PR #{entry['pr']} has no remaining change; review its closure before publishing") + parent = entry['head'] + print(f"#{entry['pr']}: {parent[:8]} - {'reused verified tree' if entry['validation']['reused'] else 'checks passed'}", flush=True) + proof = simulate_squash_order(repo, base, entries) + combined_tree = preview_tree(repo, parent, active_overlays(manifest)) + plan = dict(base=base, entries=entries, squash_merge_proof=proof, preview_tree=combined_tree, + pi_rebuild_needed=combined_tree != manifest.get('preview', {}).get('server_tree')) + write(state/'prepared.json', plan) + print('Stack merge order verified. Pi rebuild needed: '+str(plan['pi_rebuild_needed']), flush=True) + if mode == 'refresh': + return + else: + plan = json.loads((state/'prepared.json').read_text(encoding='utf-8')) + if plan['base'] != base or [e['pr'] for e in plan['entries']] != [e['pr'] for e in entries]: + raise RuntimeError('Upstream or stack state changed; refresh before publishing') + pushes = [] + leases = [] + for entry in plan['entries']: + path = local_path(entry['worktree']) + if git(path, 'status', '--porcelain').stdout or git(path, 'rev-parse', 'HEAD').stdout.strip() != entry['head']: + raise RuntimeError('Prepared candidate changed; refresh before publishing') + if entry['head'] != entry['remote_head']: + leases.append(f"--force-with-lease=refs/heads/{entry['branch']}:{entry['remote_head']}") + pushes.append(f"{entry['head']}:refs/heads/{entry['branch']}") + verify_publish_state(manifest, plan) + if pushes: + git(repo, 'push', '--atomic', *leases, manifest['remote'], *pushes) + for entry in plan['entries']: + actual = git(repo, 'ls-remote', manifest['remote'], 'refs/heads/'+entry['branch']).stdout.split()[0] + if actual != entry['head']: + raise RuntimeError('Published branch verification failed') + entry['remote_head'] = actual + guard(entry['worktree'], 'Finish', state/f"finish-{entry['pr']}.json") + manifest['entries'] = plan['entries'] + write(manifest_path, manifest) + print(('Published changed branches atomically.' if pushes else 'No branch updates were needed.') + ' Check GitHub CI before merge; no upstream merges were performed.') + + +if __name__ == '__main__': + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('mode', choices=('status', 'start', 'refresh', 'publish', 'sync', 'check')) + parser.add_argument('--project', choices=('server', 'web'), default='server') + parser.add_argument('--workspace', type=Path, help='Workspace containing planning, evidence and isolated worktrees') + parser.add_argument('--manifest', type=Path, help='Explicit stack manifest; repository/worktree paths may be workspace-relative') + parser.add_argument('--state', type=Path, help='Local validation and prepared-state directory') + parser.add_argument('--branch', help='New feature branch for Start') + parser.add_argument('--guard-script', help='Additional PowerShell repository guard; missing/failing guards stop work') + try: + args = parser.parse_args() + if args.workspace: + ROOT = args.workspace.resolve() + if args.guard_script: + GUARD_SCRIPT = args.guard_script + main(args.mode, args.manifest or ROOT/'planning'/('review-stack-web.json' if args.project == 'web' else 'review-stack.json'), + args.state or ROOT/'evidence'/('review-stack-web' if args.project == 'web' else 'review-stack'), args.branch) + except (RuntimeError, OSError, ValueError) as error: + print(str(error), file=sys.stderr) + sys.exit(1) diff --git a/tools/review-stack-web.example.json b/tools/review-stack-web.example.json new file mode 100644 index 0000000..e5e4eb9 --- /dev/null +++ b/tools/review-stack-web.example.json @@ -0,0 +1,10 @@ +{ + "kind": "web", + "repo": "beacon-web", + "upstream": "MeshCore-Beacon/beacon-web", + "fork": "YOUR_ACCOUNT/beacon-web-contributions", + "remote": "contribution", + "allowed_skips": ["Analyze"], + "preview_overlays": [], + "entries": [] +} diff --git a/tools/review-stack.example.json b/tools/review-stack.example.json new file mode 100644 index 0000000..4f753c9 --- /dev/null +++ b/tools/review-stack.example.json @@ -0,0 +1,7 @@ +{ + "repo": "beacon-server", + "upstream": "MeshCore-Beacon/beacon-server", + "fork": "YOUR_ACCOUNT/beacon-server-contributions", + "remote": "contribution", + "entries": [] +} diff --git a/tools/test_beacon_stack.py b/tools/test_beacon_stack.py new file mode 100644 index 0000000..7c0f4a6 --- /dev/null +++ b/tools/test_beacon_stack.py @@ -0,0 +1,171 @@ +"""Offline regression checks for the local stack helper; no GitHub/Pi access.""" +import json +from pathlib import Path +import tempfile +import unittest +from unittest.mock import patch + +import beacon_stack as stack + + +class StackTests(unittest.TestCase): + def test_missing_configured_guard_stops_before_commands(self): + with tempfile.TemporaryDirectory(prefix='beacon-missing-guard-') as directory: + root = Path(directory); manifest = root/'stack.json' + stack.write(manifest, dict(guard_script=str(root/'missing.ps1'))) + with patch.object(stack, 'GUARD_SCRIPT', None), patch.object(stack, 'command', side_effect=AssertionError('must stop before commands')): + with self.assertRaisesRegex(RuntimeError, 'guard is missing'): + stack.main('start', manifest, root/'state', branch='codex/next') + + def test_publication_rechecks_upstream_after_validation(self): + manifest = dict(upstream='MeshCore-Beacon/beacon-web', fork='example/beacon-web', entries=[]) + class Result: + stdout = 'new-upstream' + with patch.object(stack, 'command', return_value=Result()): + with self.assertRaisesRegex(RuntimeError, 'changed during validation'): + stack.verify_publish_state(manifest, dict(base='old-upstream', entries=[])) + stack.verify_publish_state(manifest, dict(base='new-upstream', entries=[])) + + def test_portable_guard_rejects_default_and_dirty_branches(self): + with tempfile.TemporaryDirectory(prefix='beacon-guard-test-') as directory: + repo = Path(directory); self.init_repo(repo); self.commit(repo, 'file.txt', 'base\n') + stack.git(repo, 'checkout', '-B', 'dev') + with patch.object(stack, 'GUARD_SCRIPT', None): + with self.assertRaisesRegex(RuntimeError, 'isolated feature branch'): + stack.guard(repo, 'Preflight', repo/'receipt.json') + stack.git(repo, 'checkout', '-b', 'codex/feature') + (repo/'file.txt').write_text('dirty\n') + with self.assertRaisesRegex(RuntimeError, 'isolated feature branch'): + stack.guard(repo, 'Preflight', repo/'receipt.json') + + def test_start_after_all_merges_uses_current_dev_without_rebase_or_build(self): + with tempfile.TemporaryDirectory(prefix='beacon-start-test-') as directory: + root = Path(directory); repo = root/'repo'; repo.mkdir(); self.init_repo(repo) + old = self.commit(repo, 'file.txt', 'old\n') + current = self.commit(repo, 'file.txt', 'accepted\n') + stack.git(repo, 'remote', 'add', 'origin', 'https://github.com/MeshCore-Beacon/beacon-web.git') + stack.git(repo, 'remote', 'add', 'contribution', 'https://github.com/example/beacon-web.git') + stack.git(repo, 'update-ref', 'refs/remotes/origin/dev', current) + manifest = root/'stack.json' + stack.write(manifest, dict(repo='repo', upstream='MeshCore-Beacon/beacon-web', fork='example/beacon-web', remote='contribution', entries=[dict(pr=1, branch='codex/old', head=old, remote_head=old, base=old, worktree='repo')])) + original = stack.command + class Result: + returncode = 0 + stdout = '' + def fake_command(args, **kwargs): + if args[0] == 'gh': + r = Result() + r.stdout = current if 'commits/dev' in args[2] else json.dumps(dict(merged=True, head=dict(repo=dict(full_name='example/beacon-web')))) + return r + if args[:2] == ['git', 'fetch']: + return Result() + if args[0] in ('npm', 'node', 'go', 'swag') or args[:2] == ['git', 'rebase']: + raise AssertionError('A fresh phase must not rebase or build') + return original(args, **kwargs) + with patch.object(stack, 'ROOT', root), patch.object(stack, 'GUARD_SCRIPT', None), patch.object(stack, 'command', fake_command): + stack.main('start', manifest, root/'state', branch='codex/next') + started = json.loads((root/'state/started.json').read_text()) + self.assertEqual(started['parent'], current) + self.assertIsNone(started['depends_on']) + self.assertEqual(stack.git(started['worktree'], 'rev-parse', 'HEAD').stdout.strip(), current) + + def test_remote_updates_and_closed_prs_stop_publication(self): + entries = [dict(pr=1, branch='codex/one', remote_head='1'*40, head='1'*40, base='2'*40)] + pull = dict(merged=False, state='open', head=dict(sha='1'*40, ref='codex/one'), base=dict(ref='dev')) + self.assertEqual(len(stack.active_entries(entries, [pull])), 1) + for changed in [dict(pull, state='closed'), dict(pull, head=dict(sha='someone-elses-work', ref='codex/one'))]: + with self.assertRaises(RuntimeError): + stack.active_entries(entries, [changed]) + self.assertEqual(stack.active_entries(entries, [dict(pull, merged=True, state='closed')]), []) + with self.assertRaisesRegex(RuntimeError, 'exact commit IDs'): + stack.active_entries([dict(entries[0], base='--exec=untrusted')], [pull]) + + def test_squash_merges_then_drop_accepted_parent(self): + with tempfile.TemporaryDirectory(prefix='beacon-stack-test-') as directory: + repo = Path(directory) + self.init_repo(repo) + base = self.commit(repo, 'settings.go', 'base\n') + first = self.commit(repo, 'settings.go', 'base\naccounts\n') + second = self.commit(repo, 'settings.go', 'base\naccounts\nbackup\n') + entries = [dict(pr=1, head=first, base=base), dict(pr=2, head=second, base=first)] + proof = stack.simulate_squash_order(repo, base, entries) + self.assertTrue(all(row['clean_after_squash_and_refresh'] for row in proof)) + tree = stack.git(repo, 'rev-parse', first+'^{tree}').stdout.strip() + squash = stack.git(repo, 'commit-tree', tree, '-p', base, '-m', 'accepted first PR').stdout.strip() + stack.git(repo, 'checkout', '-b', 'child', second) + stack.rebase(repo, squash, first) + self.assertEqual(stack.git(repo, 'rev-parse', 'HEAD^{tree}').stdout.strip(), proof[-1]['tree']) + self.assertEqual(stack.git(repo, 'rev-list', '--count', squash+'..HEAD').stdout.strip(), '1') + + def test_source_conflict_is_left_for_review(self): + with tempfile.TemporaryDirectory(prefix='beacon-stack-test-') as directory: + repo = Path(directory) + self.init_repo(repo) + base = self.commit(repo, 'main.go', 'base\n') + child = self.commit(repo, 'main.go', 'our change\n') + stack.git(repo, 'checkout', '-b', 'upstream', base) + upstream = self.commit(repo, 'main.go', 'different upstream change\n') + stack.git(repo, 'checkout', '-b', 'candidate', child) + with self.assertRaisesRegex(RuntimeError, 'Source conflict preserved'): + stack.rebase(repo, upstream, base) + self.assertEqual(stack.git(repo, 'diff', '--name-only', '--diff-filter=U').stdout.strip(), 'main.go') + self.assertEqual(stack.git(repo, 'rev-parse', 'refs/heads/candidate').stdout.strip(), child) + stack.git(repo, 'rebase', '--abort') + + def test_verified_tree_skips_builds(self): + with tempfile.TemporaryDirectory(prefix='beacon-stack-cache-') as directory: + cache = Path(directory) + stack.write(cache/'tree.json', dict(passed=True, tools='go / swag', build_environment='{}')) + class Result: + stdout = '' + def fake_git(repo, *args, **kwargs): + result = Result() + result.stdout = 'tree\n' if args[0] == 'rev-parse' else '' + return result + def fake_command(args, **kwargs): + if args[:2] == ['go', 'env']: + result = Result(); result.stdout = '{}'; return result + if args not in (['go', 'version'], ['swag', '--version']): + raise AssertionError('cached source tree unexpectedly rebuilt') + result = Result(); result.stdout = args[0] + return result + with patch.object(stack, 'git', fake_git), patch.object(stack, 'command', fake_command): + self.assertTrue(stack.validate(cache, cache)['reused']) + + def test_web_environment_fingerprint_changes_without_recording_values(self): + with tempfile.TemporaryDirectory(prefix='beacon-web-cache-') as directory: + root = Path(directory) + first = stack.web_fingerprint(root) + (root/'.env.local').write_text('VITE_FIXTURE=PRIVATE_CANARY\n', encoding='utf-8') + second = stack.web_fingerprint(root) + self.assertNotEqual(first, second) + self.assertNotIn('PRIVATE_CANARY', second) + self.assertEqual(len(second), 64) + + def test_preview_overlay_preserves_both_independent_changes(self): + with tempfile.TemporaryDirectory(prefix='beacon-preview-test-') as directory: + repo = Path(directory); self.init_repo(repo) + base = self.commit(repo, 'base.txt', 'base\n') + feature = self.commit(repo, 'analytics.txt', 'charts\n') + stack.git(repo, 'checkout', '-b', 'overlay', base) + overlay = self.commit(repo, 'nodes.txt', 'badges\n') + tree = stack.preview_tree(repo, feature, [overlay]) + self.assertEqual(set(stack.git(repo, 'ls-tree', '--name-only', tree).stdout.splitlines()), {'base.txt', 'analytics.txt', 'nodes.txt'}) + + @staticmethod + def init_repo(repo): + stack.git(repo, 'init', '-q') + stack.git(repo, 'config', 'user.name', 'Stack test') + stack.git(repo, 'config', 'user.email', 'test@example.invalid') + stack.git(repo, 'config', 'core.autocrlf', 'false') + + @staticmethod + def commit(repo, file, text): + (repo/file).write_text(text, encoding='utf-8', newline='\n') + stack.git(repo, 'add', file) + stack.git(repo, 'commit', '-q', '-m', text.strip().replace('\n', ' ')) + return stack.git(repo, 'rev-parse', 'HEAD').stdout.strip() + + +if __name__ == '__main__': + unittest.main() From 911ed74a2af7dca60af5d87f2a364bd73e548349 Mon Sep 17 00:00:00 2001 From: n30nex Date: Sun, 20 Sep 2026 00:15:32 -0400 Subject: [PATCH 02/69] ci(workflow): check contributor branches as well as pull requests --- .github/workflows/contributor-tools.yml | 1 - CONTRIBUTOR_WORKFLOW.md | 2 +- 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/.github/workflows/contributor-tools.yml b/.github/workflows/contributor-tools.yml index 73f8a1e..b37fb76 100644 --- a/.github/workflows/contributor-tools.yml +++ b/.github/workflows/contributor-tools.yml @@ -4,7 +4,6 @@ on: pull_request: paths: ["tools/**", ".github/workflows/contributor-tools.yml"] push: - branches: [main] paths: ["tools/**", ".github/workflows/contributor-tools.yml"] permissions: diff --git a/CONTRIBUTOR_WORKFLOW.md b/CONTRIBUTOR_WORKFLOW.md index 7f38c90..d7c0614 100644 --- a/CONTRIBUTOR_WORKFLOW.md +++ b/CONTRIBUTOR_WORKFLOW.md @@ -1,6 +1,6 @@ # Contributor workflow -Use this workflow to keep small Beacon changes reviewable while reducing manual branch maintenance. The executable helper is [tools/beacon_stack.py](tools/beacon_stack.py); its [offline regression tests](tools/test_beacon_stack.py) run in this repository's CI. +Use this workflow to keep small Beacon changes reviewable while reducing manual branch maintenance. The executable helper is [tools/beacon_stack.py](tools/beacon_stack.py); a CI workflow is included for its [offline regression tests](tools/test_beacon_stack.py). ## Working loop From ce53377eaff44218254b799f1ccb7eb1adf05c03 Mon Sep 17 00:00:00 2001 From: n30nex Date: Sun, 20 Sep 2026 13:06:35 -0400 Subject: [PATCH 03/69] docs(release): freeze the current queue and record review validation --- CONTRIBUTOR_WORKFLOW.md | 5 +++- RELEASE-CHECKLIST.md | 51 +++++++++++++++++++++++++++++++++++++++++ ROADMAP.md | 20 ++++++++-------- 3 files changed, 66 insertions(+), 10 deletions(-) create mode 100644 RELEASE-CHECKLIST.md diff --git a/CONTRIBUTOR_WORKFLOW.md b/CONTRIBUTOR_WORKFLOW.md index d7c0614..6f32a54 100644 --- a/CONTRIBUTOR_WORKFLOW.md +++ b/CONTRIBUTOR_WORKFLOW.md @@ -9,7 +9,8 @@ Use this workflow to keep small Beacon changes reviewable while reducing manual 3. Follow each repository's contribution rules. Prefer existing components and feature-owned files; keep shared startup/router/navigation changes in a declared order. 4. Build and test the change, open its PR against `dev`, then record its exact parent, head, fork branch and worktree in the manifest. Link the parent-to-head comparison in the PR body. 5. Keep current Beacon contribution PRs out of draft as requested by the contributor, with dependencies visible. Request MrAlders0n's review; if account permissions prevent formal assignment, use an explicit review-request comment instead. -6. After merges, run Sync and Check. Rebuild/redeploy the preview only if the composed source changes, preserving the actual running revision and corresponding-source offer. +6. Keep cross-repository rollout order explicit: a new web page waits until its server endpoint is merged **and deployed**. Ready for review does not imply ready to merge. +7. After merges, run Sync and Check. Rebuild/redeploy the preview only if the composed source changes, preserving the actual running revision and corresponding-source offer. A source conflict still needs review. The helper automates routine history movement; it does not promise that overlapping edits can never conflict, merge upstream PRs, deploy services, or create scheduled jobs. @@ -56,6 +57,8 @@ An entry records the feature's delta boundary, not a guessed merge base: `remote_head` is the lease protecting someone else's newer work. Never overwrite it to suppress a mismatch. Review external changes before updating the manifest. +If a maintainer lands several stacked PRs in one squash and closes the included parent, the helper deliberately stops. Verify the closed head is an ancestor of the accepted child, compare that child's tree with the upstream squash, and retain the maintainer's disposition link. Then remove only the verified included entry from the active manifest and run Sync. Do not reopen it or override a closed-state check without that evidence. Web #52 included by #53 is the first recorded example. + ## What Sync actually does - Reads current `dev` and PR state, drops merged parents and rejects unexpectedly changed or unmerged-closed PRs. diff --git a/RELEASE-CHECKLIST.md b/RELEASE-CHECKLIST.md new file mode 100644 index 0000000..d418ce1 --- /dev/null +++ b/RELEASE-CHECKLIST.md @@ -0,0 +1,51 @@ +# Server/web consolidation release + +Status: preparation, not a published release or a full CoreScope parity claim. Updated 20 September 2026 after maintainer review of the four server PRs. + +## Scope and stop point + +Finish the current account/backup/analytics queue, correct review regressions, and release that bounded set before Channel Activity or further parity expansion. Current public tags are server v1.6.0 and web v1.3.0; maintainers choose the next versions and perform signed release commits, main promotion and tags under each repository's contribution rules. + +The deployment owner performs the eventual CoreScope switch. Beacon remains at dev.meshcore.ca, CoreScope at live.meshcore.ca, and the Pi preview remains at canadaverse.org/beacon-dev/ with its changelog and corresponding source. + +## Review gates + +- [x] Server #149: document POST/DELETE browser preflights and the full admin CORS method example. Keep public read-only defaults. +- [x] Server #154: verify pg_dump/server compatibility at startup; an optional backup prerequisite failure disables only backup, with a specific operator diagnostic. Document backup.enabled and distinguish the export size limit. Native testing caught and fixed the text-versus-integer version-setting scan; CI now covers it with PostgreSQL. +- [x] Server #157: serve Signal distributions and weighted means from compact materialized data; snap polling windows to hours, preserve missing/invalid/legacy sample semantics, and measure refresh/storage costs. +- [x] Server #159: materialize path classification, share window parsing, guard database-derived array indexes and retain all 256 decoder-header checks. +- [x] Web #58: distinct navigation glyphs in #59, plus dedicated RF/Signal and Paths glyphs in #55/#57. +- [x] Refresh #55/#57 after the accepted #52/#53 squash. Their source trees were identical after the September 20 refresh; that history-only update needs no replacement Pi artifact. +- [x] All seven application PRs pass their required checks on the published heads. Native PostgreSQL tests ran. The upstream web CodeQL job remains skipped under its existing policy and is not counted as a scan. + +These checkmarks record completed corrections and validation, not maintainer approval. The four server PRs, three web PRs and docs #5 still await their merge/review decisions. + +Review order is server #149 -> #154 -> #157 -> #159. Web #55 waits for #157 to be merged **and deployed**; #57 waits for #159 to be merged and deployed and for #55. The icon fix can be reviewed independently. Traffic and Scopes have already landed together as web e01c090; do not replay #52. + +## Storage and retention boundary + +The September 17 drop-and-reset observation-partitioning design and implementation plan were explicitly superseded on September 19. They are historical reference only. Do not implement their table drop, history reset or process-local dedup replacement. + +The stated replacement direction is lz4 compression, batched deletes, per-table autovacuum tuning and a seven-day default. Those changes are not present in the verified published server dev 951b79b (its example still says 30 days). Obtain and review the replacement contribution before describing it as shipped. Coordinate append-only migration numbers with that work; the old plan's proposed 035 is not evidence that a migration exists. + +A consolidation release must document its actual retention behavior and capacity limits. A future production parity cutover also needs verified durable history coverage and recovery copies. Do not infer either from example configuration or the Pi's short history. + +## Combined-candidate evidence + +- [x] Native Pi build/test of server `6be0f762` and web `19672038`, including real PostgreSQL-to-HTTP checks and migration retry/concurrent refresh; 781 web tests pass. +- [x] One-million-row request/initial-population/refresh/storage measurements; request plans read only the new views. Signal: 1.8–77.5 ms reads, 14.4 s initial population, 16.8 s refresh, 6.5 MB. Paths: 3.5–141.9 ms reads, 8.4 s population, 6.2 s refresh, 11.3 MB. +- [ ] Measure the full operator workload and sustained refresh/ingest load before a production parity claim. The million-row fixture does not establish that limit. +- [x] Backup client mismatch and unsupported-DSN cases leave the public API available; valid client export/restore used disposable data only and restored all 35 source migrations. Public preview admin/backup remains disabled. +- [x] Real preview analytics reconcile with SQL for the materialized window. Browser charts, complete-hour text, small screens and error/empty/retry states are verified. Both MQTT feeds advance and the public browser reports LIVE. +- [x] Current and rollback server/web artifacts, exact source offers and visible changelog match the running pair. Only the Beacon app restarted; the other 20 containers were preserved. Additive rollup migrations retain observations and are compatible with the previous binary. + +The rollback pair is server `4f6679c8` / web `b1100972`. The local deployment record preserves its binaries, assets, source archives and runner. Rolling back the application keeps the additive rollup views; it does not remove history. + +## Maintainer release handoff + +1. Merge the reviewed queue in the declared order, using the shared refresh helper after accepted parents. Do not manually rebase every PR from scratch. +2. Deploy the accepted server before merging/deploying pages that need its new endpoints. Verify endpoint availability on the intended deployment, not just the Pi preview. +3. Freeze exact reviewed server/web dev heads and rerun required checks on them. Refresh the release notes with only changes actually included. +4. Follow the server contribution guide for a signed version/Swagger commit, dev-to-main fast-forward, tag and release CI. Web main has a prior release squash (`5ac36ce`) outside dev ancestry; maintainers must reconcile that stable history before choosing its promotion method. Do not silently overwrite main. +5. Verify the tag's Actions-built artifacts and matching source. Publish accurate release notes, upgrade/retention guidance, known gaps and rollback instructions. +6. After both releases, empty the accepted review queues and start the next feature directly from freshly fetched dev. Resume the parity roadmap; this milestone does not close partial #60/#72, #99, #116 or internationalization #12. diff --git a/ROADMAP.md b/ROADMAP.md index 2fa43df..089bddd 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -34,14 +34,15 @@ Server order: **#149 → #154 → #157 → #159**. | [Server #157](https://github.com/MeshCore-Beacon/beacon-server/pull/157) | Bounded SNR/RSSI distributions and hourly statistics | Complete endpoint scope in #156 | | [Server #159](https://github.com/MeshCore-Beacon/beacon-server/pull/159) | Bounded received-path and hash-width statistics | Complete endpoint scope in #158 | -Web order: **#52 → #53 → #55 → #57**. Observer comparison [#48](https://github.com/MeshCore-Beacon/beacon-web/pull/48) and foreign-node display [#49](https://github.com/MeshCore-Beacon/beacon-web/pull/49) are merged; issue #26 is closed. +Web order: **#59 → #55 → #57**. The icon correction #59 can land independently; #55/#57 wait until their server endpoints are merged and deployed. Traffic #52 landed verbatim in the #53 squash (`e01c090`); #52 was closed as included, and #53 is merged. Issues #50/#51 are closed. Observer comparison [#48](https://github.com/MeshCore-Beacon/beacon-web/pull/48) and foreign-node display [#49](https://github.com/MeshCore-Beacon/beacon-web/pull/49) are also merged. | PR | Scope | Dependency / issue | |---|---|---| -| [Web #52](https://github.com/MeshCore-Beacon/beacon-web/pull/52) | Traffic heatmap, hourly trends and reception share | #50 | -| [Web #53](https://github.com/MeshCore-Beacon/beacon-web/pull/53) | Regional scope charts and exact counts | After #52; #51 | -| [Web #55](https://github.com/MeshCore-Beacon/beacon-web/pull/55) | RF / Signal charts and sample availability | After #53 and server #157; #54 | -| [Web #57](https://github.com/MeshCore-Beacon/beacon-web/pull/57) | Paths & Hashes charts and classification coverage | After #55 and server #159; #56 | +| [Web #52](https://github.com/MeshCore-Beacon/beacon-web/pull/52) | Traffic heatmap, hourly trends and reception share | Included in merged #53; #50 closed | +| [Web #53](https://github.com/MeshCore-Beacon/beacon-web/pull/53) | Regional scope charts and exact counts | Merged as e01c090; #51 closed | +| [Web #59](https://github.com/MeshCore-Beacon/beacon-web/pull/59) | Distinct Traffic, Scopes and Compare icons | Independent correction; #58 | +| [Web #55](https://github.com/MeshCore-Beacon/beacon-web/pull/55) | RF / Signal charts and sample availability | After #59 and server #157 is merged and deployed; #54 | +| [Web #57](https://github.com/MeshCore-Beacon/beacon-web/pull/57) | Paths & Hashes charts and classification coverage | After #55 and server #159 is merged and deployed; #56 | The router foundation [server #155](https://github.com/MeshCore-Beacon/beacon-server/pull/155) is merged. The refresh workflow drops accepted squash parents, preserves focused feature deltas and handles changed branch history in isolated worktrees. See [the executable contributor workflow](CONTRIBUTOR_WORKFLOW.md). @@ -63,14 +64,14 @@ The router foundation [server #155](https://github.com/MeshCore-Beacon/beacon-se The path page counts stored receptions, not unique devices. Flood paths accumulate entries, while direct routes carry remaining entries. Observed widths do not establish device capability or collision rates. Signal readings describe reception at the reporting observer rather than a complete end-to-end link. -Both new aggregate APIs accept explicit windows of at most 30 days, propagate cancellation, use region-specific caching and have a 15-second execution bound. A raw 30-day global query can still take seconds at larger volumes. On a shared Pi with one million synthetic rows, signal queries measured about 2.1–6.5 seconds and path queries 2.8–7.5 seconds across prepared-plan modes. Measure production volume before choosing precomputed distributions or claiming release readiness. +The September 20 review correction moves both new aggregate APIs onto materialized hourly snapshots, preserving reception/region semantics and normalizing polling windows to UTC hours. In a rolled-back million-row Pi fixture, Signal request queries took 1.8–77.5 ms and Paths 3.5–141.9 ms across custom/generic plans. Initial population took 14.4/8.4 seconds, refresh 16.8/6.2 seconds, and view/index storage was 6.5/11.3 MB respectively. These measurements cover the fixture, not the full production workload. See the [release consolidation checklist](RELEASE-CHECKLIST.md) for remaining gates. -The current combined preview has passed native Go/PostgreSQL/HTTP validation and **781 web tests**, plus desktop/mobile layout checks at 320/390/768/1280px. Synthetic fixtures tested gaps, trace/unclassified data, zero samples, errors/retry and long windows; they were not published. Current revisions and corresponding-source archives are on the preview's changelog page. +The current combined preview has passed native Go/PostgreSQL/HTTP validation and **781 web tests**. Private backup checks cover version compatibility, feature-only startup failure, TLS/password files, cancellation and schema/data/sequence restoration. The review update passed 390/1280px browser checks, with ten distinct glyphs and explicit complete-hour text; earlier chart validation also covered 320/768px. Public Signal/Paths counts reconcile with SQL, both MQTT feeds advance and the browser reports LIVE. Current revisions and corresponding-source archives are on the preview's changelog page. ## Next phases -1. **Review feedback and regressions first.** Refresh both application queues, resolve attributable failures, and retain explicit issue scope. -2. **Channel Activity analytics.** Inspect existing aggregates and add bounded channel traffic/trend views. Define messages versus receptions, unknown/encrypted-channel coverage and the behavior when keys are unavailable. Keep keys and message contents out of aggregate responses; avoid raw-message paging to build statistics. +1. **Finish the current queue and prepare a consolidation release.** Address all four server reviews and the analytics icon regression [web #58](https://github.com/MeshCore-Beacon/beacon-web/issues/58). Validate the combined candidate on the Pi, retain backend-before-frontend deployment order, then hand reviewed `dev` candidates to maintainers for the server/web main releases. Follow [RELEASE-CHECKLIST.md](RELEASE-CHECKLIST.md). Pause new analytics until this breakpoint; this release does not claim complete CoreScope parity. +2. **After the consolidation release: Channel Activity analytics.** Inspect existing aggregates and add bounded channel traffic/trend views. Define messages versus receptions, unknown/encrypted-channel coverage and the behavior when keys are unavailable. Keep keys and message contents out of aggregate responses; avoid raw-message paging to build statistics. 3. **Observed ambiguity and topology.** Separate static prefix conflicts from observed unresolved/ambiguous paths. Add justified route-pattern, neighbour, hop and distance views with clear provenance and bounded work. 4. **Queued administration and backup slices.** Add non-destructive archive validation, then resolve browser login/session, import and deployment-file coverage separately. Test restoration against disposable databases. 5. **Production evidence and handoff.** Reconcile history, operational limits and the parity matrix below before preparing a release/cutover handoff. @@ -84,6 +85,7 @@ The current combined preview has passed native Go/PostgreSQL/HTTP validation and | [Server #60](https://github.com/MeshCore-Beacon/beacon-server/issues/60) | Remaining administration/worker/persistence behavior; account records do not establish login sessions | | [Server #72](https://github.com/MeshCore-Beacon/beacon-server/issues/72) | Archive validation/import, browser access, deployment-file coverage and remote/scheduled backup scope | | [Web #12](https://github.com/MeshCore-Beacon/beacon-web/issues/12) | Supported-language and formatting scope for internationalization | +| [Web #58](https://github.com/MeshCore-Beacon/beacon-web/issues/58) | Distinct analytics navigation icons, including pending RF/Signal and Paths pages | The analytics endpoint/page issues remain open while their corresponding PRs await merge. Use closing references only when a PR completes the issue's accepted scope; use related references for partial work. From 9da0867fe374863720e1ad47e00e392b6b85d916 Mon Sep 17 00:00:00 2001 From: n30nex Date: Sun, 20 Sep 2026 14:18:31 -0400 Subject: [PATCH 04/69] docs(roadmap): record the map location regression fix --- RELEASE-CHECKLIST.md | 9 +++++---- ROADMAP.md | 8 ++++++-- 2 files changed, 11 insertions(+), 6 deletions(-) diff --git a/RELEASE-CHECKLIST.md b/RELEASE-CHECKLIST.md index d418ce1..3afc07b 100644 --- a/RELEASE-CHECKLIST.md +++ b/RELEASE-CHECKLIST.md @@ -14,11 +14,12 @@ The deployment owner performs the eventual CoreScope switch. Beacon remains at d - [x] Server #154: verify pg_dump/server compatibility at startup; an optional backup prerequisite failure disables only backup, with a specific operator diagnostic. Document backup.enabled and distinguish the export size limit. Native testing caught and fixed the text-versus-integer version-setting scan; CI now covers it with PostgreSQL. - [x] Server #157: serve Signal distributions and weighted means from compact materialized data; snap polling windows to hours, preserve missing/invalid/legacy sample semantics, and measure refresh/storage costs. - [x] Server #159: materialize path classification, share window parsing, guard database-derived array indexes and retain all 256 decoder-header checks. +- [x] Web #60: shared map-location validation in independent #61 omits reset/invalid markers and links while preserving valid zero-axis locations and stored records. Native and real-data browser checks pass. - [x] Web #58: distinct navigation glyphs in #59, plus dedicated RF/Signal and Paths glyphs in #55/#57. - [x] Refresh #55/#57 after the accepted #52/#53 squash. Their source trees were identical after the September 20 refresh; that history-only update needs no replacement Pi artifact. -- [x] All seven application PRs pass their required checks on the published heads. Native PostgreSQL tests ran. The upstream web CodeQL job remains skipped under its existing policy and is not counted as a scan. +- [x] All eight application PRs pass their required checks on the published heads. Native PostgreSQL tests ran. The upstream web CodeQL job remains skipped under its existing policy and is not counted as a scan. -These checkmarks record completed corrections and validation, not maintainer approval. The four server PRs, three web PRs and docs #5 still await their merge/review decisions. +These checkmarks record completed corrections and validation, not maintainer approval. The four server PRs, four web PRs and docs #5 still await their merge/review decisions. Review order is server #149 -> #154 -> #157 -> #159. Web #55 waits for #157 to be merged **and deployed**; #57 waits for #159 to be merged and deployed and for #55. The icon fix can be reviewed independently. Traffic and Scopes have already landed together as web e01c090; do not replay #52. @@ -32,14 +33,14 @@ A consolidation release must document its actual retention behavior and capacity ## Combined-candidate evidence -- [x] Native Pi build/test of server `6be0f762` and web `19672038`, including real PostgreSQL-to-HTTP checks and migration retry/concurrent refresh; 781 web tests pass. +- [x] Native Pi build/test of server `6be0f762` and web `42ba5fc`, including real PostgreSQL-to-HTTP checks and migration retry/concurrent refresh; 786 web tests pass. - [x] One-million-row request/initial-population/refresh/storage measurements; request plans read only the new views. Signal: 1.8–77.5 ms reads, 14.4 s initial population, 16.8 s refresh, 6.5 MB. Paths: 3.5–141.9 ms reads, 8.4 s population, 6.2 s refresh, 11.3 MB. - [ ] Measure the full operator workload and sustained refresh/ingest load before a production parity claim. The million-row fixture does not establish that limit. - [x] Backup client mismatch and unsupported-DSN cases leave the public API available; valid client export/restore used disposable data only and restored all 35 source migrations. Public preview admin/backup remains disabled. - [x] Real preview analytics reconcile with SQL for the materialized window. Browser charts, complete-hour text, small screens and error/empty/retry states are verified. Both MQTT feeds advance and the public browser reports LIVE. - [x] Current and rollback server/web artifacts, exact source offers and visible changelog match the running pair. Only the Beacon app restarted; the other 20 containers were preserved. Additive rollup migrations retain observations and are compatible with the previous binary. -The rollback pair is server `4f6679c8` / web `b1100972`. The local deployment record preserves its binaries, assets, source archives and runner. Rolling back the application keeps the additive rollup views; it does not remove history. +The immediate frontend rollback is web `19672038` with the unchanged server `6be0f762`. The earlier full rollback pair, server `4f6679c8` / web `b1100972`, is also retained. The local deployment record preserves its binaries, assets, source archives and runner. Rolling back the application keeps the additive rollup views; it does not remove history. ## Maintainer release handoff diff --git a/ROADMAP.md b/ROADMAP.md index 089bddd..7356df3 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -34,12 +34,13 @@ Server order: **#149 → #154 → #157 → #159**. | [Server #157](https://github.com/MeshCore-Beacon/beacon-server/pull/157) | Bounded SNR/RSSI distributions and hourly statistics | Complete endpoint scope in #156 | | [Server #159](https://github.com/MeshCore-Beacon/beacon-server/pull/159) | Bounded received-path and hash-width statistics | Complete endpoint scope in #158 | -Web order: **#59 → #55 → #57**. The icon correction #59 can land independently; #55/#57 wait until their server endpoints are merged and deployed. Traffic #52 landed verbatim in the #53 squash (`e01c090`); #52 was closed as included, and #53 is merged. Issues #50/#51 are closed. Observer comparison [#48](https://github.com/MeshCore-Beacon/beacon-web/pull/48) and foreign-node display [#49](https://github.com/MeshCore-Beacon/beacon-web/pull/49) are also merged. +Web order: **#59 → #55 → #57**, with independent map correction **#61** also ready. The icon correction #59 can land independently; #55/#57 wait until their server endpoints are merged and deployed. Traffic #52 landed verbatim in the #53 squash (`e01c090`); #52 was closed as included, and #53 is merged. Issues #50/#51 are closed. Observer comparison [#48](https://github.com/MeshCore-Beacon/beacon-web/pull/48) and foreign-node display [#49](https://github.com/MeshCore-Beacon/beacon-web/pull/49) are also merged. | PR | Scope | Dependency / issue | |---|---|---| | [Web #52](https://github.com/MeshCore-Beacon/beacon-web/pull/52) | Traffic heatmap, hourly trends and reception share | Included in merged #53; #50 closed | | [Web #53](https://github.com/MeshCore-Beacon/beacon-web/pull/53) | Regional scope charts and exact counts | Merged as e01c090; #51 closed | +| [Web #61](https://github.com/MeshCore-Beacon/beacon-web/pull/61) | Omit reset/invalid map locations and false neighbour/path lines | Independent correction; #60 | | [Web #59](https://github.com/MeshCore-Beacon/beacon-web/pull/59) | Distinct Traffic, Scopes and Compare icons | Independent correction; #58 | | [Web #55](https://github.com/MeshCore-Beacon/beacon-web/pull/55) | RF / Signal charts and sample availability | After #59 and server #157 is merged and deployed; #54 | | [Web #57](https://github.com/MeshCore-Beacon/beacon-web/pull/57) | Paths & Hashes charts and classification coverage | After #55 and server #159 is merged and deployed; #56 | @@ -66,7 +67,9 @@ The path page counts stored receptions, not unique devices. Flood paths accumula The September 20 review correction moves both new aggregate APIs onto materialized hourly snapshots, preserving reception/region semantics and normalizing polling windows to UTC hours. In a rolled-back million-row Pi fixture, Signal request queries took 1.8–77.5 ms and Paths 3.5–141.9 ms across custom/generic plans. Initial population took 14.4/8.4 seconds, refresh 16.8/6.2 seconds, and view/index storage was 6.5/11.3 MB respectively. These measurements cover the fixture, not the full production workload. See the [release consolidation checklist](RELEASE-CHECKLIST.md) for remaining gates. -The current combined preview has passed native Go/PostgreSQL/HTTP validation and **781 web tests**. Private backup checks cover version compatibility, feature-only startup failure, TLS/password files, cancellation and schema/data/sequence restoration. The review update passed 390/1280px browser checks, with ten distinct glyphs and explicit complete-hour text; earlier chart validation also covered 320/768px. Public Signal/Paths counts reconcile with SQL, both MQTT feeds advance and the browser reports LIVE. Current revisions and corresponding-source archives are on the preview's changelog page. +The current combined preview has passed native Go/PostgreSQL/HTTP validation and **786 web tests**. Private backup checks cover version compatibility, feature-only startup failure, TLS/password files, cancellation and schema/data/sequence restoration. The review update passed 390/1280px browser checks, with ten distinct glyphs and explicit complete-hour text; earlier chart validation also covered 320/768px. Public Signal/Paths counts reconcile with SQL, both MQTT feeds advance and the browser reports LIVE. Current revisions and corresponding-source archives are on the preview's changelog page. + +The current Pi web build is `42ba5fc` with server `6be0f762`. The map correction preserves stored records and valid equator/prime-meridian positions; browser checks with real data confirm the false origin cluster and Atlantic links are gone with neighbour lines enabled. All service containers remained unchanged during this frontend-only update. ## Next phases @@ -85,6 +88,7 @@ The current combined preview has passed native Go/PostgreSQL/HTTP validation and | [Server #60](https://github.com/MeshCore-Beacon/beacon-server/issues/60) | Remaining administration/worker/persistence behavior; account records do not establish login sessions | | [Server #72](https://github.com/MeshCore-Beacon/beacon-server/issues/72) | Archive validation/import, browser access, deployment-file coverage and remote/scheduled backup scope | | [Web #12](https://github.com/MeshCore-Beacon/beacon-web/issues/12) | Supported-language and formatting scope for internationalization | +| [Web #60](https://github.com/MeshCore-Beacon/beacon-web/issues/60) | Map location-reset correction #61 is deployed and awaiting merge | | [Web #58](https://github.com/MeshCore-Beacon/beacon-web/issues/58) | Distinct analytics navigation icons, including pending RF/Signal and Paths pages | The analytics endpoint/page issues remain open while their corresponding PRs await merge. Use closing references only when a PR completes the issue's accepted scope; use related references for partial work. From 8f50b6cbadcfbd562437aadc5ef4129d4cb691bd Mon Sep 17 00:00:00 2001 From: n30nex Date: Sun, 20 Sep 2026 15:23:04 -0400 Subject: [PATCH 05/69] docs(roadmap): record independent backup archive verification --- RELEASE-CHECKLIST.md | 2 ++ ROADMAP.md | 7 +++++-- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/RELEASE-CHECKLIST.md b/RELEASE-CHECKLIST.md index 3afc07b..fc59d84 100644 --- a/RELEASE-CHECKLIST.md +++ b/RELEASE-CHECKLIST.md @@ -8,6 +8,8 @@ Finish the current account/backup/analytics queue, correct review regressions, a The deployment owner performs the eventual CoreScope switch. Beacon remains at dev.meshcore.ca, CoreScope at live.meshcore.ca, and the Pi preview remains at canadaverse.org/beacon-dev/ with its changelog and corresponding source. +Independent follow-up [server #160](https://github.com/MeshCore-Beacon/beacon-server/pull/160) adds offline archive verification and does not block or expand this release's required queue. Its separately installed Pi CLI is `9385bc14`; the running server/web stay unchanged. Required native and compiled PostgreSQL checks pass, with Windows race coverage because the Pi race runtime cannot initialize. Decide explicitly whether to include the CLI follow-up when freezing the release; issue #72 remains partial either way. + ## Review gates - [x] Server #149: document POST/DELETE browser preflights and the full admin CORS method example. Keep public read-only defaults. diff --git a/ROADMAP.md b/ROADMAP.md index 7356df3..68399aa 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -33,6 +33,9 @@ Server order: **#149 → #154 → #157 → #159**. | [Server #154](https://github.com/MeshCore-Beacon/beacon-server/pull/154) | Protected database/config backup download | Partial #72; login, validation/import and deployment-file coverage remain | | [Server #157](https://github.com/MeshCore-Beacon/beacon-server/pull/157) | Bounded SNR/RSSI distributions and hourly statistics | Complete endpoint scope in #156 | | [Server #159](https://github.com/MeshCore-Beacon/beacon-server/pull/159) | Bounded received-path and hash-width statistics | Complete endpoint scope in #158 | +| [Server #160](https://github.com/MeshCore-Beacon/beacon-server/pull/160) | Offline, non-destructive backup archive verification | Independent accepted-dev parent `951b79b`; partial #72, outside the shared stack | + +#160 is an independent CLI follow-up and does not block the consolidation release. Candidate `9385bc14` passes Windows/native Pi build, format, vet and unit checks, Windows race checks, malformed-input/fuzz cases, and exact compiled PostgreSQL export/verify/restore; CI/CodeQL pass. The Pi race runtime cannot initialize on its kernel and is recorded as unavailable. The CLI is installed separately, with its own source link on the preview changelog; server `6be0f762` / web `42ba5fc` and all 21 existing containers were preserved. The validator checks archive structure and integrity, not authenticity, safe SQL or restorability. Web order: **#59 → #55 → #57**, with independent map correction **#61** also ready. The icon correction #59 can land independently; #55/#57 wait until their server endpoints are merged and deployed. Traffic #52 landed verbatim in the #53 squash (`e01c090`); #52 was closed as included, and #53 is merged. Issues #50/#51 are closed. Observer comparison [#48](https://github.com/MeshCore-Beacon/beacon-web/pull/48) and foreign-node display [#49](https://github.com/MeshCore-Beacon/beacon-web/pull/49) are also merged. @@ -76,7 +79,7 @@ The current Pi web build is `42ba5fc` with server `6be0f762`. The map correction 1. **Finish the current queue and prepare a consolidation release.** Address all four server reviews and the analytics icon regression [web #58](https://github.com/MeshCore-Beacon/beacon-web/issues/58). Validate the combined candidate on the Pi, retain backend-before-frontend deployment order, then hand reviewed `dev` candidates to maintainers for the server/web main releases. Follow [RELEASE-CHECKLIST.md](RELEASE-CHECKLIST.md). Pause new analytics until this breakpoint; this release does not claim complete CoreScope parity. 2. **After the consolidation release: Channel Activity analytics.** Inspect existing aggregates and add bounded channel traffic/trend views. Define messages versus receptions, unknown/encrypted-channel coverage and the behavior when keys are unavailable. Keep keys and message contents out of aggregate responses; avoid raw-message paging to build statistics. 3. **Observed ambiguity and topology.** Separate static prefix conflicts from observed unresolved/ambiguous paths. Add justified route-pattern, neighbour, hop and distance views with clear provenance and bounded work. -4. **Queued administration and backup slices.** Add non-destructive archive validation, then resolve browser login/session, import and deployment-file coverage separately. Test restoration against disposable databases. +4. **Administration and backup slices.** Non-destructive archive validation is ready in independent #160. Resolve browser login/session and restore authorization before an import endpoint; keep deployment-file coverage and remote/scheduled backup separate. Test restoration against disposable databases. These follow-ups do not delay the current release. 5. **Production evidence and handoff.** Reconcile history, operational limits and the parity matrix below before preparing a release/cutover handoff. ## Listed work still open @@ -86,7 +89,7 @@ The current Pi web build is `42ba5fc` with server `6be0f762`. The map correction | [Server #116](https://github.com/MeshCore-Beacon/beacon-server/issues/116) | Attribute repeated MQTT ping timeouts to a measured cause; fresh traffic alone is insufficient | | [Server #99](https://github.com/MeshCore-Beacon/beacon-server/issues/99) | Define and complete remaining packet-type summary coverage after advert summaries | | [Server #60](https://github.com/MeshCore-Beacon/beacon-server/issues/60) | Remaining administration/worker/persistence behavior; account records do not establish login sessions | -| [Server #72](https://github.com/MeshCore-Beacon/beacon-server/issues/72) | Archive validation/import, browser access, deployment-file coverage and remote/scheduled backup scope | +| [Server #72](https://github.com/MeshCore-Beacon/beacon-server/issues/72) | Archive validation #160 awaits review; import, browser access, deployment-file coverage and remote/scheduled backup remain | | [Web #12](https://github.com/MeshCore-Beacon/beacon-web/issues/12) | Supported-language and formatting scope for internationalization | | [Web #60](https://github.com/MeshCore-Beacon/beacon-web/issues/60) | Map location-reset correction #61 is deployed and awaiting merge | | [Web #58](https://github.com/MeshCore-Beacon/beacon-web/issues/58) | Distinct analytics navigation icons, including pending RF/Signal and Paths pages | From 8255e4ddc31fba903259972970be142926087d03 Mon Sep 17 00:00:00 2001 From: n30nex Date: Sun, 20 Sep 2026 15:32:35 -0400 Subject: [PATCH 06/69] docs(roadmap): pin the conflict-free verifier candidate --- RELEASE-CHECKLIST.md | 2 +- ROADMAP.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/RELEASE-CHECKLIST.md b/RELEASE-CHECKLIST.md index fc59d84..87fd073 100644 --- a/RELEASE-CHECKLIST.md +++ b/RELEASE-CHECKLIST.md @@ -8,7 +8,7 @@ Finish the current account/backup/analytics queue, correct review regressions, a The deployment owner performs the eventual CoreScope switch. Beacon remains at dev.meshcore.ca, CoreScope at live.meshcore.ca, and the Pi preview remains at canadaverse.org/beacon-dev/ with its changelog and corresponding source. -Independent follow-up [server #160](https://github.com/MeshCore-Beacon/beacon-server/pull/160) adds offline archive verification and does not block or expand this release's required queue. Its separately installed Pi CLI is `9385bc14`; the running server/web stay unchanged. Required native and compiled PostgreSQL checks pass, with Windows race coverage because the Pi race runtime cannot initialize. Decide explicitly whether to include the CLI follow-up when freezing the release; issue #72 remains partial either way. +Independent follow-up [server #160](https://github.com/MeshCore-Beacon/beacon-server/pull/160) adds offline archive verification and does not block or expand this release's required queue. Its separately installed Pi CLI is `262eae96`; the running server/web stay unchanged. Required native and compiled PostgreSQL checks pass, with Windows race coverage because the Pi race runtime cannot initialize. Decide explicitly whether to include the CLI follow-up when freezing the release; issue #72 remains partial either way. ## Review gates diff --git a/ROADMAP.md b/ROADMAP.md index 68399aa..25496d9 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -35,7 +35,7 @@ Server order: **#149 → #154 → #157 → #159**. | [Server #159](https://github.com/MeshCore-Beacon/beacon-server/pull/159) | Bounded received-path and hash-width statistics | Complete endpoint scope in #158 | | [Server #160](https://github.com/MeshCore-Beacon/beacon-server/pull/160) | Offline, non-destructive backup archive verification | Independent accepted-dev parent `951b79b`; partial #72, outside the shared stack | -#160 is an independent CLI follow-up and does not block the consolidation release. Candidate `9385bc14` passes Windows/native Pi build, format, vet and unit checks, Windows race checks, malformed-input/fuzz cases, and exact compiled PostgreSQL export/verify/restore; CI/CodeQL pass. The Pi race runtime cannot initialize on its kernel and is recorded as unavailable. The CLI is installed separately, with its own source link on the preview changelog; server `6be0f762` / web `42ba5fc` and all 21 existing containers were preserved. The validator checks archive structure and integrity, not authenticity, safe SQL or restorability. +#160 is an independent CLI follow-up and does not block the consolidation release. Candidate `262eae96` passes Windows/native Pi build, format, vet and unit checks, Windows race checks, malformed-input/fuzz cases, and exact compiled PostgreSQL export/verify/restore; CI/CodeQL pass. The Pi race runtime cannot initialize on its kernel and is recorded as unavailable. The CLI is installed separately, with its own source link on the preview changelog; server `6be0f762` / web `42ba5fc` and all 21 existing containers were preserved. The validator checks archive structure and integrity, not authenticity, safe SQL or restorability. Web order: **#59 → #55 → #57**, with independent map correction **#61** also ready. The icon correction #59 can land independently; #55/#57 wait until their server endpoints are merged and deployed. Traffic #52 landed verbatim in the #53 squash (`e01c090`); #52 was closed as included, and #53 is merged. Issues #50/#51 are closed. Observer comparison [#48](https://github.com/MeshCore-Beacon/beacon-web/pull/48) and foreign-node display [#49](https://github.com/MeshCore-Beacon/beacon-web/pull/49) are also merged. From 34b98be1d3edde59178ea45a132ca6159e5363b6 Mon Sep 17 00:00:00 2001 From: n30nex Date: Sun, 20 Sep 2026 16:08:45 -0400 Subject: [PATCH 07/69] docs(roadmap): record packet-reference summary delivery --- RELEASE-CHECKLIST.md | 4 +++- ROADMAP.md | 9 ++++++--- 2 files changed, 9 insertions(+), 4 deletions(-) diff --git a/RELEASE-CHECKLIST.md b/RELEASE-CHECKLIST.md index 87fd073..7aa0965 100644 --- a/RELEASE-CHECKLIST.md +++ b/RELEASE-CHECKLIST.md @@ -10,6 +10,8 @@ The deployment owner performs the eventual CoreScope switch. Beacon remains at d Independent follow-up [server #160](https://github.com/MeshCore-Beacon/beacon-server/pull/160) adds offline archive verification and does not block or expand this release's required queue. Its separately installed Pi CLI is `262eae96`; the running server/web stay unchanged. Required native and compiled PostgreSQL checks pass, with Windows race coverage because the Pi race runtime cannot initialize. Decide explicitly whether to include the CLI follow-up when freezing the release; issue #72 remains partial either way. +Independent follow-up [server #161](https://github.com/MeshCore-Beacon/beacon-server/pull/161), `0befdc5c`, adds ACK/TRACE/PING summaries in the existing packet display and also leaves the release queue independent. It merges cleanly with the reviewed stack. Combined server `5848d200` is on the Pi with web `42ba5fcb`; native PostgreSQL, Windows race, CI and public REST/live/browser checks pass. No schema/config changes or extra public admin access. Include it only if accepted when freezing; #99 remains partial for other requested formats. + ## Review gates - [x] Server #149: document POST/DELETE browser preflights and the full admin CORS method example. Keep public read-only defaults. @@ -42,7 +44,7 @@ A consolidation release must document its actual retention behavior and capacity - [x] Real preview analytics reconcile with SQL for the materialized window. Browser charts, complete-hour text, small screens and error/empty/retry states are verified. Both MQTT feeds advance and the public browser reports LIVE. - [x] Current and rollback server/web artifacts, exact source offers and visible changelog match the running pair. Only the Beacon app restarted; the other 20 containers were preserved. Additive rollup migrations retain observations and are compatible with the previous binary. -The immediate frontend rollback is web `19672038` with the unchanged server `6be0f762`. The earlier full rollback pair, server `4f6679c8` / web `b1100972`, is also retained. The local deployment record preserves its binaries, assets, source archives and runner. Rolling back the application keeps the additive rollup views; it does not remove history. +The immediate server rollback from the packet-reference update is `6be0f762` with current web `42ba5fcb`. The previous frontend rollback `19672038` and earlier full pair, server `4f6679c8` / web `b1100972`, are also retained. The local deployment record preserves binaries, assets, source archives and runners. Rolling back the application keeps the additive rollup views; it does not remove history. ## Maintainer release handoff diff --git a/ROADMAP.md b/ROADMAP.md index 25496d9..a5498ce 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -34,8 +34,11 @@ Server order: **#149 → #154 → #157 → #159**. | [Server #157](https://github.com/MeshCore-Beacon/beacon-server/pull/157) | Bounded SNR/RSSI distributions and hourly statistics | Complete endpoint scope in #156 | | [Server #159](https://github.com/MeshCore-Beacon/beacon-server/pull/159) | Bounded received-path and hash-width statistics | Complete endpoint scope in #158 | | [Server #160](https://github.com/MeshCore-Beacon/beacon-server/pull/160) | Offline, non-destructive backup archive verification | Independent accepted-dev parent `951b79b`; partial #72, outside the shared stack | +| [Server #161](https://github.com/MeshCore-Beacon/beacon-server/pull/161) | ACK identifiers and TRACE/PING tags in packet summaries | Independent accepted-dev parent `951b79b`; partial #99, clean overlay on the shared stack | -#160 is an independent CLI follow-up and does not block the consolidation release. Candidate `262eae96` passes Windows/native Pi build, format, vet and unit checks, Windows race checks, malformed-input/fuzz cases, and exact compiled PostgreSQL export/verify/restore; CI/CodeQL pass. The Pi race runtime cannot initialize on its kernel and is recorded as unavailable. The CLI is installed separately, with its own source link on the preview changelog; server `6be0f762` / web `42ba5fc` and all 21 existing containers were preserved. The validator checks archive structure and integrity, not authenticity, safe SQL or restorability. +#160 is an independent CLI follow-up and does not block the consolidation release. Candidate `262eae96` passes Windows/native Pi build, format, vet and unit checks, Windows race checks, malformed-input/fuzz cases, and exact compiled PostgreSQL export/verify/restore; CI/CodeQL pass. The Pi race runtime cannot initialize on its kernel and is recorded as unavailable. The CLI remains installed separately, with its own source link on the preview changelog. The validator checks archive structure and integrity, not authenticity, safe SQL or restorability. + +#161 at `0befdc5c` also remains independent. Native builds of the standalone and combined candidates pass, including 28 PostgreSQL fixtures that keep each global/regional/backfill read to one query; Windows ingest race checks and CI/CodeQL pass. Public historical summaries and a live PING match packet detail, with 1280px/390px browser checks and no horizontal overflow. The existing web renders the new text without a client release. References are packet-carried checksums/tags, not identities or delivery/authentication guarantees. No message body or trace auth code is added. This follow-up does not block the consolidation release. Web order: **#59 → #55 → #57**, with independent map correction **#61** also ready. The icon correction #59 can land independently; #55/#57 wait until their server endpoints are merged and deployed. Traffic #52 landed verbatim in the #53 squash (`e01c090`); #52 was closed as included, and #53 is merged. Issues #50/#51 are closed. Observer comparison [#48](https://github.com/MeshCore-Beacon/beacon-web/pull/48) and foreign-node display [#49](https://github.com/MeshCore-Beacon/beacon-web/pull/49) are also merged. @@ -72,7 +75,7 @@ The September 20 review correction moves both new aggregate APIs onto materializ The current combined preview has passed native Go/PostgreSQL/HTTP validation and **786 web tests**. Private backup checks cover version compatibility, feature-only startup failure, TLS/password files, cancellation and schema/data/sequence restoration. The review update passed 390/1280px browser checks, with ten distinct glyphs and explicit complete-hour text; earlier chart validation also covered 320/768px. Public Signal/Paths counts reconcile with SQL, both MQTT feeds advance and the browser reports LIVE. Current revisions and corresponding-source archives are on the preview's changelog page. -The current Pi web build is `42ba5fc` with server `6be0f762`. The map correction preserves stored records and valid equator/prime-meridian positions; browser checks with real data confirm the false origin cluster and Atlantic links are gone with neighbour lines enabled. All service containers remained unchanged during this frontend-only update. +The current Pi build is server `5848d200` with unchanged web `42ba5fcb`. It composes #161 with the reviewed server stack; no existing PR branch was rewritten. Both MQTT feeds advance, schema/config remain unchanged and the other 20 containers were preserved. Packet-reference reads took 4.39–22.83 ms at the origin. The map correction and separate backup-verifier tool remain included. Current/rollback source and binaries are retained; only verified inactive staging was retired. ## Next phases @@ -87,7 +90,7 @@ The current Pi web build is `42ba5fc` with server `6be0f762`. The map correction | Issue | Remaining scope | |---|---| | [Server #116](https://github.com/MeshCore-Beacon/beacon-server/issues/116) | Attribute repeated MQTT ping timeouts to a measured cause; fresh traffic alone is insufficient | -| [Server #99](https://github.com/MeshCore-Beacon/beacon-server/issues/99) | Define and complete remaining packet-type summary coverage after advert summaries | +| [Server #99](https://github.com/MeshCore-Beacon/beacon-server/issues/99) | Advert names are accepted; ACK/TRACE/PING references in #161 await review; define any remaining packet-type formats | | [Server #60](https://github.com/MeshCore-Beacon/beacon-server/issues/60) | Remaining administration/worker/persistence behavior; account records do not establish login sessions | | [Server #72](https://github.com/MeshCore-Beacon/beacon-server/issues/72) | Archive validation #160 awaits review; import, browser access, deployment-file coverage and remote/scheduled backup remain | | [Web #12](https://github.com/MeshCore-Beacon/beacon-web/issues/12) | Supported-language and formatting scope for internationalization | From 2b9e0804aa4f9e2eef43c77b63fed78e683faa8b Mon Sep 17 00:00:00 2001 From: n30nex Date: Sun, 20 Sep 2026 19:51:46 -0400 Subject: [PATCH 08/69] docs(roadmap): record current MQTT stability measurements --- RELEASE-CHECKLIST.md | 1 + ROADMAP.md | 4 +++- 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/RELEASE-CHECKLIST.md b/RELEASE-CHECKLIST.md index 7aa0965..7831e26 100644 --- a/RELEASE-CHECKLIST.md +++ b/RELEASE-CHECKLIST.md @@ -37,6 +37,7 @@ A consolidation release must document its actual retention behavior and capacity ## Combined-candidate evidence +- [x] September 20 unmodified Pi stability sample: 600 seconds / 41 samples, both feeds connected, 2,169 retained observations and no MQTT disconnect/deadline or HTTP 5xx. App/PostgreSQL CPU averaged 2.14%/3.96% of one core. This is a bounded health sample, not callback timing, #116 root-cause proof or a production-volume gate. [Result and limits](https://github.com/MeshCore-Beacon/beacon-server/issues/116#issuecomment-5753626821). - [x] Native Pi build/test of server `6be0f762` and web `42ba5fc`, including real PostgreSQL-to-HTTP checks and migration retry/concurrent refresh; 786 web tests pass. - [x] One-million-row request/initial-population/refresh/storage measurements; request plans read only the new views. Signal: 1.8–77.5 ms reads, 14.4 s initial population, 16.8 s refresh, 6.5 MB. Paths: 3.5–141.9 ms reads, 8.4 s population, 6.2 s refresh, 11.3 MB. - [ ] Measure the full operator workload and sustained refresh/ingest load before a production parity claim. The million-row fixture does not establish that limit. diff --git a/ROADMAP.md b/ROADMAP.md index a5498ce..4f53370 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -79,6 +79,8 @@ The current Pi build is server `5848d200` with unchanged web `42ba5fcb`. It comp ## Next phases +The September 20 #116 investigation has a new [current-build result](https://github.com/MeshCore-Beacon/beacon-server/issues/116#issuecomment-5753626821): a 600-second unmodified Pi capture kept both feeds connected and retained 2,169 new observations, with no ping timeout, disconnect, deadline, SQLSTATE error or HTTP 5xx response. App/PostgreSQL CPU averaged 2.14%/3.96% of one core. The preceding 3h39 log likewise has no MQTT loss or deadline error. Timestamp warnings were classified separately. This did not measure callback or pool-acquisition duration and does not establish the original cause or production capacity. No application, ordering, acknowledgement or service change was made; #116 remains open. Further capture should follow a recurrence or meaningful workload change, rather than repeatedly sampling the same healthy state. + 1. **Finish the current queue and prepare a consolidation release.** Address all four server reviews and the analytics icon regression [web #58](https://github.com/MeshCore-Beacon/beacon-web/issues/58). Validate the combined candidate on the Pi, retain backend-before-frontend deployment order, then hand reviewed `dev` candidates to maintainers for the server/web main releases. Follow [RELEASE-CHECKLIST.md](RELEASE-CHECKLIST.md). Pause new analytics until this breakpoint; this release does not claim complete CoreScope parity. 2. **After the consolidation release: Channel Activity analytics.** Inspect existing aggregates and add bounded channel traffic/trend views. Define messages versus receptions, unknown/encrypted-channel coverage and the behavior when keys are unavailable. Keep keys and message contents out of aggregate responses; avoid raw-message paging to build statistics. 3. **Observed ambiguity and topology.** Separate static prefix conflicts from observed unresolved/ambiguous paths. Add justified route-pattern, neighbour, hop and distance views with clear provenance and bounded work. @@ -89,7 +91,7 @@ The current Pi build is server `5848d200` with unchanged web `42ba5fcb`. It comp | Issue | Remaining scope | |---|---| -| [Server #116](https://github.com/MeshCore-Beacon/beacon-server/issues/116) | Attribute repeated MQTT ping timeouts to a measured cause; fresh traffic alone is insufficient | +| [Server #116](https://github.com/MeshCore-Beacon/beacon-server/issues/116) | No recurrence in the September 20 retained log / ten-minute capture; still needs an attributable event with callback/pool timing | | [Server #99](https://github.com/MeshCore-Beacon/beacon-server/issues/99) | Advert names are accepted; ACK/TRACE/PING references in #161 await review; define any remaining packet-type formats | | [Server #60](https://github.com/MeshCore-Beacon/beacon-server/issues/60) | Remaining administration/worker/persistence behavior; account records do not establish login sessions | | [Server #72](https://github.com/MeshCore-Beacon/beacon-server/issues/72) | Archive validation #160 awaits review; import, browser access, deployment-file coverage and remote/scheduled backup remain | From 1aaf0a5bd0ae60ef68b22af6f6564266912c9c0c Mon Sep 17 00:00:00 2001 From: n30nex Date: Sun, 20 Sep 2026 20:49:26 -0400 Subject: [PATCH 09/69] fix(workflow): validate independent preview PRs and prepared inputs --- CONTRIBUTOR_WORKFLOW.md | 10 ++- RELEASE-CHECKLIST.md | 1 + ROADMAP.md | 2 + tools/beacon_stack.py | 45 ++++++++--- tools/test_beacon_stack.py | 156 +++++++++++++++++++++++++++++++++++++ 5 files changed, 201 insertions(+), 13 deletions(-) diff --git a/CONTRIBUTOR_WORKFLOW.md b/CONTRIBUTOR_WORKFLOW.md index 6f32a54..1c42152 100644 --- a/CONTRIBUTOR_WORKFLOW.md +++ b/CONTRIBUTOR_WORKFLOW.md @@ -68,7 +68,7 @@ If a maintainer lands several stacked PRs in one squash and closes the included - Rechecks upstream and PR state immediately before publishing, including after a long validation run. If a merge happened meanwhile, it stops and retains reusable receipts. - Publishes changed fork branches atomically with explicit leases, verifies remote heads and reports current CI separately. -`refresh` prepares without pushing; `publish` publishes a prepared result after fresh checks. `sync` combines both. `check` requires current published heads and passing required checks; only explicitly configured skipped jobs are allowed. +`refresh` prepares without pushing; `publish` publishes a prepared result after fresh checks. `sync` combines both. `check` requires current published heads and passing required checks for the ordered queue **and every active independent preview PR**; only explicitly configured skipped jobs are allowed. A passing build is required for each. The helper rechecks source/merge state after reading CI and stops if it observes a changed head or merge state. Local validation reuse does not suppress GitHub's checks on a changed head. The first refresh after an independent change joins several candidates can require new combination checks; later identical-tree refreshes reuse those receipts. @@ -76,7 +76,11 @@ Local validation reuse does not suppress GitHub's checks on a changed head. The Set `preview.server_tree` to the verified composed Git tree only after native/public validation. This field is also used in the web manifest for compatibility. A different commit with an identical tree does not require rebuilding or relabeling an existing artifact. Keep the real built revision/source archive. -Independent pending work can be listed in `preview_overlays` as `{"pr": 123, "head": "EXACT_COMMIT"}`. Merged overlays drop automatically; changed or unmerged-closed overlays stop composition. Keep overlapping changes in the main ordered queue. +Independent pending work can be listed in `preview_overlays` as `{"pr": 123, "head": "EXACT_COMMIT"}`. Use full 40-character commit IDs and list each PR only once, outside the ordered queue. Status lists the active overlays; Check includes their CI results, marked `preview_overlay: true`. Merged overlays drop automatically; changed, unmerged-closed, wrong-fork or wrong-target overlays stop the workflow. Keep overlapping changes in the main ordered queue. + +Refresh records the exact independent inputs in its prepared plan and verifies them after local validation. Publish rechecks them immediately before pushing. If an overlay moved, merged or was added/removed after preparation, refresh again; unchanged source still reuses validation. Prepared files made by older helper versions without an overlay snapshot must be refreshed when active overlays exist. This does not require rebasing unrelated PRs or rebuilding an identical preview. + +Legacy bare commit overlays can still be composed, but Status marks them unverified and Check refuses to report a complete green result without a PR/head record. Standalone tools that are not part of the app preview, such as the backup-verifier CLI, retain their separate manifests/checks. After all queued changes merge, Sync leaves an empty queue. Start then creates the next branch directly from current `dev`. This is the normal path for a new phase, without carrying historical feature commits forward. @@ -92,4 +96,4 @@ The Canadaverse workspace keeps its required site guard in the local wrapper. Pu python -m unittest discover -s tools -p test_beacon_stack.py -v ``` -Tests cover squash/drop-parent behavior, a fresh phase after all merges, cache reuse, environment changes, independent overlays, dirty/default-branch rejection, source conflicts and upstream movement during validation. They use disposable local Git repositories and no GitHub or Pi credentials. +Tests cover squash/drop-parent behavior, a fresh phase after all merges, cache reuse, environment changes, independent overlay CI/state/identity checks, missing or pending checks, skipped-job policy, publication races, dirty/default-branch rejection, source conflicts and upstream movement during validation. They use disposable local Git repositories and no GitHub or Pi credentials. diff --git a/RELEASE-CHECKLIST.md b/RELEASE-CHECKLIST.md index 7831e26..b5090e0 100644 --- a/RELEASE-CHECKLIST.md +++ b/RELEASE-CHECKLIST.md @@ -14,6 +14,7 @@ Independent follow-up [server #161](https://github.com/MeshCore-Beacon/beacon-se ## Review gates +- [x] Workflow checks include every independent preview PR, not just the ordered stack. Status reports them; Check verifies their CI and source; Refresh/Publish reject changed prepared inputs. Server #161 and web #61 are covered by the normal preview checks. The standalone backup CLI #160 is checked with its separate manifest. - [x] Server #149: document POST/DELETE browser preflights and the full admin CORS method example. Keep public read-only defaults. - [x] Server #154: verify pg_dump/server compatibility at startup; an optional backup prerequisite failure disables only backup, with a specific operator diagnostic. Document backup.enabled and distinguish the export size limit. Native testing caught and fixed the text-versus-integer version-setting scan; CI now covers it with PostgreSQL. - [x] Server #157: serve Signal distributions and weighted means from compact materialized data; snap polling windows to hours, preserve missing/invalid/legacy sample semantics, and measure refresh/storage costs. diff --git a/ROADMAP.md b/ROADMAP.md index 4f53370..01d5c8c 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -4,6 +4,8 @@ Updated 20 September 2026. This is the working roadmap for n30nex's ongoing cont Refresh GitHub issues, PR feedback and branch state before starting a phase. This document is a snapshot, and linked issues/PRs are the current source of truth. +Release-check correction, 21 September UTC: the workflow now includes independent preview PRs in Status and Check, applies the same CI/head/fork/target requirements to them, and rechecks the prepared independent inputs before publication. This covers packet summaries #161 and map correction #61 without adding them to the ordered stacks. The backup CLI #160 retains its separate check. Twenty-three offline regressions cover these gates and the existing no-rebase/cache behavior. See [the contributor workflow](CONTRIBUTOR_WORKFLOW.md). + ## Direction Bring useful CoreScope investigation and analytics features into Beacon's existing ingest, database, API, cache and web components. Prioritize review regressions and measured stability/performance problems, then useful analytics pages. Keep each API or page a focused contribution with explicit counting semantics and validation. diff --git a/tools/beacon_stack.py b/tools/beacon_stack.py index dcd749d..7771090 100644 --- a/tools/beacon_stack.py +++ b/tools/beacon_stack.py @@ -106,6 +106,8 @@ def verify_publish_state(manifest, plan): active = active_entries(manifest['entries'], pulls) if plan['base'] != base or [e['pr'] for e in plan['entries']] != [e['pr'] for e in active]: raise RuntimeError('Upstream or merge state changed during validation; refresh again. Verified trees remain cached.') + if plan.get('preview_overlays', []) != active_overlays(manifest): + raise RuntimeError('Independent preview overlays changed during validation; refresh again. Verified trees remain cached.') def rebase(path, parent, previous_base, kind='server'): @@ -199,18 +201,30 @@ def preview_tree(repo, head, overlays): def active_overlays(manifest): - heads = [] + active = [] + seen = {entry['pr'] for entry in manifest['entries']} for overlay in manifest.get('preview_overlays', []): if isinstance(overlay, str): - heads.append(overlay) + if not re.fullmatch(r'[a-f0-9]{40}', overlay): + raise RuntimeError('Independent preview boundaries must be exact commit IDs') + active.append(dict(pr=None, head=overlay)) continue + if not isinstance(overlay, dict) or type(overlay.get('pr')) is not int or overlay['pr'] <= 0: + raise RuntimeError('Independent preview PRs need a positive PR number') + if not isinstance(overlay.get('head'), str) or not re.fullmatch(r'[a-f0-9]{40}', overlay['head']): + raise RuntimeError('Independent preview boundaries must be exact commit IDs') + if overlay['pr'] in seen: + raise RuntimeError('An independent preview PR is listed more than once or also in the ordered stack') + seen.add(overlay['pr']) pull = json.loads(command(['gh', 'api', f"repos/{manifest['upstream']}/pulls/{overlay['pr']}"]).stdout) + if pull['base']['ref'] != 'dev': + raise RuntimeError('Unexpected independent preview PR target; this workflow only targets dev') if pull['merged']: continue if pull['state'] != 'open' or pull['head']['sha'] != overlay['head'] or pull['head']['repo']['full_name'] != manifest['fork']: - raise RuntimeError('An independent preview candidate changed; review it before composing the preview') - heads.append(overlay['head']) - return heads + raise RuntimeError(f"An independent preview candidate (PR #{overlay['pr']}) changed; review it before continuing") + active.append(dict(pr=overlay['pr'], head=overlay['head'])) + return active def simulate_squash_order(repo, base, entries): @@ -250,6 +264,7 @@ def main(mode, manifest_path=MANIFEST, state=STATE, branch=None): if pull['head']['repo']['full_name'] != manifest['fork']: raise RuntimeError('Unexpected fork identity') entries = active_entries(manifest['entries'], pulls) + overlays = active_overlays(manifest) base = command(['gh', 'api', f"repos/{manifest['upstream']}/commits/dev", '--jq', '.sha']).stdout.strip() parent = base needs_refresh = False @@ -259,14 +274,19 @@ def main(mode, manifest_path=MANIFEST, state=STATE, branch=None): if mode == 'check': if needs_refresh: raise RuntimeError('The stack has an outdated parent; refresh before relying on previous checks') + if any(overlay['pr'] is None for overlay in overlays): + raise RuntimeError('Independent preview commits need a PR/head record before GitHub checks can be verified') results = [] - for entry in entries: + targets = [(entry, False) for entry in entries] + [(dict(overlay, remote_head=overlay['head']), True) for overlay in overlays] + for entry, is_overlay in targets: result = command(['gh', 'pr', 'checks', str(entry['pr']), '-R', manifest['upstream'], '--json', 'name,state,bucket,link'], check=False) checks = json.loads(result.stdout) if result.stdout.lstrip().startswith('[') else [] - passed = bool(checks) and all(c['bucket'] == 'pass' or (c['bucket'] == 'skipping' and c['name'] in manifest.get('allowed_skips', [])) for c in checks) and any(c['name'] == 'build' and c['bucket'] == 'pass' for c in checks) - results.append(dict(pr=entry['pr'], head=entry['remote_head'], + passed = result.returncode == 0 and bool(checks) and all(c['bucket'] == 'pass' or (c['bucket'] == 'skipping' and c['name'] in manifest.get('allowed_skips', [])) for c in checks) and any(c['name'] == 'build' and c['bucket'] == 'pass' for c in checks) + results.append(dict(pr=entry['pr'], head=entry['remote_head'], preview_overlay=is_overlay, source_verified=entry['head'] == entry['remote_head'], passed=passed, checks=checks)) + # A PR-number check can otherwise race with a new push or accepted merge. + verify_publish_state(manifest, dict(base=base, entries=entries, preview_overlays=overlays)) write(state/'github-checks.json', results) print(json.dumps(results, indent=2)) if not all(r['source_verified'] and r['passed'] for r in results): @@ -274,6 +294,8 @@ def main(mode, manifest_path=MANIFEST, state=STATE, branch=None): return if mode == 'status': print(json.dumps(dict(upstream=base, order=[e['pr'] for e in entries], needs_refresh=needs_refresh, + preview_overlays=overlays, + unverified_overlays=[o['head'] for o in overlays if o['pr'] is None], merged=[e['pr'] for e, p in zip(manifest['entries'], pulls) if p['merged']], unpublished=[e['pr'] for e in entries if e['head'] != e['remote_head']]), indent=2)) return @@ -337,9 +359,12 @@ def main(mode, manifest_path=MANIFEST, state=STATE, branch=None): parent = entry['head'] print(f"#{entry['pr']}: {parent[:8]} - {'reused verified tree' if entry['validation']['reused'] else 'checks passed'}", flush=True) proof = simulate_squash_order(repo, base, entries) - combined_tree = preview_tree(repo, parent, active_overlays(manifest)) - plan = dict(base=base, entries=entries, squash_merge_proof=proof, preview_tree=combined_tree, + # Refresh the independent inputs too, after potentially lengthy builds. + overlays = active_overlays(manifest) + combined_tree = preview_tree(repo, parent, [overlay['head'] for overlay in overlays]) + plan = dict(base=base, entries=entries, preview_overlays=overlays, squash_merge_proof=proof, preview_tree=combined_tree, pi_rebuild_needed=combined_tree != manifest.get('preview', {}).get('server_tree')) + verify_publish_state(manifest, plan) write(state/'prepared.json', plan) print('Stack merge order verified. Pi rebuild needed: '+str(plan['pi_rebuild_needed']), flush=True) if mode == 'refresh': diff --git a/tools/test_beacon_stack.py b/tools/test_beacon_stack.py index 7c0f4a6..b10a08b 100644 --- a/tools/test_beacon_stack.py +++ b/tools/test_beacon_stack.py @@ -1,6 +1,9 @@ """Offline regression checks for the local stack helper; no GitHub/Pi access.""" +from contextlib import redirect_stdout +import io import json from pathlib import Path +import subprocess import tempfile import unittest from unittest.mock import patch @@ -8,6 +11,159 @@ import beacon_stack as stack +class PreviewCheckTests(unittest.TestCase): + def setUp(self): + self.temp = tempfile.TemporaryDirectory(prefix='beacon-preview-check-') + self.addCleanup(self.temp.cleanup) + self.root = Path(self.temp.name) + self.manifest = dict(repo='unused', upstream='MeshCore-Beacon/beacon-web', + fork='example/beacon-web', remote='contribution', entries=[ + dict(pr=1, branch='codex/one', base='a'*40, head='b'*40, remote_head='b'*40)], + preview_overlays=[dict(pr=2, head='c'*40)]) + self.pulls = {number: dict(merged=False, state='open', base=dict(ref='dev'), + head=dict(sha=head, ref='codex/one', repo=dict(full_name='example/beacon-web'))) + for number, head in ((1, 'b'*40), (2, 'c'*40))} + self.checks = {number: [dict(name='build', state='SUCCESS', bucket='pass', link='https://example.invalid/check')] + for number in (1, 2)} + self.commands = [] + self.on_checks = lambda number: None + + def command(self, args, **kwargs): + self.commands.append(args) + if args[:2] == ['gh', 'api']: + output = 'a'*40 if args[2].endswith('/commits/dev') else json.dumps(self.pulls[int(args[2].split('/')[-1])]) + return subprocess.CompletedProcess(args, 0, output) + if args[:3] == ['gh', 'pr', 'checks']: + number = int(args[3]) + checks = self.checks[number] + code = 0 if all(c['bucket'] in ('pass', 'skipping') for c in checks) else 1 + self.on_checks(number) + return subprocess.CompletedProcess(args, code, json.dumps(checks)) + raise AssertionError(f'Unexpected external command: {args}') + + def run_main(self, mode='check'): + path = self.root/'manifest.json' + stack.write(path, self.manifest) + output = io.StringIO() + with patch.object(stack, 'GUARD_SCRIPT', None), patch.object(stack, 'check_remotes'), \ + patch.object(stack, 'command', side_effect=self.command), redirect_stdout(output): + stack.main(mode, path, self.root/'state') + return json.loads(output.getvalue()) + + def test_failed_overlay_prevents_a_green_check(self): + self.checks[2][0].update(state='FAILURE', bucket='fail') + with self.assertRaisesRegex(RuntimeError, 'incomplete/failing'): + self.run_main() + + def test_changed_overlay_stops_check(self): + self.pulls[2]['head']['sha'] = 'd'*40 + with self.assertRaisesRegex(RuntimeError, 'independent preview'): + self.run_main() + + def test_check_reports_the_overlay_and_ordered_pr(self): + rows = self.run_main() + self.assertEqual({row['pr'] for row in rows}, {1, 2}) + self.assertTrue(all(row['passed'] and row['source_verified'] for row in rows)) + self.assertEqual({row['pr'] for row in rows if row['preview_overlay']}, {2}) + + def test_pending_or_missing_overlay_checks_fail(self): + for checks in ([], [dict(name='build', state='PENDING', bucket='pending')]): + with self.subTest(checks=checks): + self.checks[2] = checks + with self.assertRaisesRegex(RuntimeError, 'incomplete/failing'): + self.run_main() + + def test_overlay_uses_the_explicit_skip_policy_and_requires_a_build(self): + self.checks[2].append(dict(name='Analyze', state='SKIPPED', bucket='skipping')) + with self.assertRaisesRegex(RuntimeError, 'incomplete/failing'): + self.run_main() + self.manifest['allowed_skips'] = ['Analyze'] + self.assertTrue(all(row['passed'] for row in self.run_main())) + self.checks[2] = self.checks[2][1:] + with self.assertRaisesRegex(RuntimeError, 'incomplete/failing'): + self.run_main() + + def test_a_failed_check_command_cannot_pass_from_partial_output(self): + original = self.command + def failed_command(args, **kwargs): + result = original(args, **kwargs) + if args[:4] == ['gh', 'pr', 'checks', '2']: + result.returncode = 1 + return result + with patch.object(self, 'command', side_effect=failed_command): + with self.assertRaisesRegex(RuntimeError, 'incomplete/failing'): + self.run_main() + + def test_merged_overlay_is_dropped_but_unmerged_closure_stops(self): + self.pulls[2].update(merged=True, state='closed') + self.assertEqual([row['pr'] for row in self.run_main()], [1]) + self.assertNotIn(['gh', 'pr', 'checks', '2'], [c[:4] for c in self.commands]) + self.pulls[2]['merged'] = False + with self.assertRaisesRegex(RuntimeError, 'independent preview'): + self.run_main() + + def test_wrong_overlay_fork_or_base_stops(self): + self.pulls[2]['head']['repo']['full_name'] = 'unexpected/beacon-web' + with self.assertRaisesRegex(RuntimeError, 'independent preview'): + self.run_main() + self.pulls[2]['head']['repo']['full_name'] = self.manifest['fork'] + self.pulls[2]['base']['ref'] = 'main' + with self.assertRaisesRegex(RuntimeError, 'target'): + self.run_main() + + def test_status_lists_independent_inputs_and_stops_on_drift(self): + result = self.run_main('status') + self.assertEqual(result['preview_overlays'], [dict(pr=2, head='c'*40)]) + self.pulls[2]['head']['sha'] = 'd'*40 + with self.assertRaisesRegex(RuntimeError, 'independent preview'): + self.run_main('status') + + def test_unlinked_commits_are_visible_but_cannot_receive_a_green_check(self): + self.manifest['preview_overlays'] = ['c'*40] + status = self.run_main('status') + self.assertEqual(status['unverified_overlays'], ['c'*40]) + with self.assertRaisesRegex(RuntimeError, 'PR/head record'): + self.run_main() + + def test_invalid_or_duplicate_overlay_records_stop_before_checks(self): + for overlays in (['--exec=untrusted'], [dict(pr=2, head='HEAD')], + [dict(pr=True, head='c'*40)], [dict(pr=2, head=None)], + [dict(pr=1, head='b'*40)], [dict(pr=2, head='c'*40)]*2): + with self.subTest(overlays=overlays): + self.manifest['preview_overlays'] = overlays + self.commands.clear() + with self.assertRaises(RuntimeError): + self.run_main() + self.assertFalse(any(c[:3] == ['gh', 'pr', 'checks'] for c in self.commands)) + + def test_a_push_during_checks_is_detected_for_both_kinds_of_pr(self): + for number in (1, 2): + with self.subTest(number=number): + original = self.pulls[number]['head']['sha'] + def push(checked): + if checked == 2: + self.pulls[number]['head']['sha'] = 'd'*40 + self.on_checks = push + with self.assertRaisesRegex(RuntimeError, 'changed'): + self.run_main() + self.pulls[number]['head']['sha'] = original + + def test_publication_requires_the_same_independent_inputs_as_preparation(self): + plan = dict(base='a'*40, entries=self.manifest['entries'], preview_overlays=[dict(pr=2, head='c'*40)]) + with patch.object(stack, 'command', side_effect=self.command): + stack.verify_publish_state(self.manifest, plan) + # Old prepared files cannot silently omit an active overlay. + with self.assertRaisesRegex(RuntimeError, 'preview overlays changed'): + stack.verify_publish_state(self.manifest, dict(base=plan['base'], entries=plan['entries'])) + self.pulls[2]['merged'] = True + with self.assertRaisesRegex(RuntimeError, 'preview overlays changed'): + stack.verify_publish_state(self.manifest, plan) + self.pulls[2]['merged'] = False + self.pulls[2]['head']['sha'] = 'd'*40 + with self.assertRaisesRegex(RuntimeError, 'independent preview'): + stack.verify_publish_state(self.manifest, plan) + + class StackTests(unittest.TestCase): def test_missing_configured_guard_stops_before_commands(self): with tempfile.TemporaryDirectory(prefix='beacon-missing-guard-') as directory: From 9ae3a4370043dc905d6192275e0f8f8b2e840431 Mon Sep 17 00:00:00 2001 From: n30nex Date: Thu, 24 Sep 2026 05:17:35 -0400 Subject: [PATCH 10/69] docs(roadmap): record accepted consolidation and Pi validation --- RELEASE-CHECKLIST.md | 41 ++++++++++++++++++------------ ROADMAP.md | 59 ++++++++++++++++++-------------------------- 2 files changed, 49 insertions(+), 51 deletions(-) diff --git a/RELEASE-CHECKLIST.md b/RELEASE-CHECKLIST.md index b5090e0..1bd9a86 100644 --- a/RELEASE-CHECKLIST.md +++ b/RELEASE-CHECKLIST.md @@ -1,16 +1,18 @@ # Server/web consolidation release -Status: preparation, not a published release or a full CoreScope parity claim. Updated 20 September 2026 after maintainer review of the four server PRs. +Status: accepted-dev validation and release handoff, 24 September 2026. All ten application/CLI PRs are merged. Stable tags remain unchanged; this is not a published release or a full CoreScope parity claim. ## Scope and stop point -Finish the current account/backup/analytics queue, correct review regressions, and release that bounded set before Channel Activity or further parity expansion. Current public tags are server v1.6.0 and web v1.3.0; maintainers choose the next versions and perform signed release commits, main promotion and tags under each repository's contribution rules. +Release the accepted account/backup/analytics batch before Channel Activity or further parity expansion. Current public tags are server v1.6.0 and web v1.3.0; maintainers choose the next versions and perform signed release commits, main promotion and tags under each repository's contribution rules. The deployment owner performs the eventual CoreScope switch. Beacon remains at dev.meshcore.ca, CoreScope at live.meshcore.ca, and the Pi preview remains at canadaverse.org/beacon-dev/ with its changelog and corresponding source. -Independent follow-up [server #160](https://github.com/MeshCore-Beacon/beacon-server/pull/160) adds offline archive verification and does not block or expand this release's required queue. Its separately installed Pi CLI is `262eae96`; the running server/web stay unchanged. Required native and compiled PostgreSQL checks pass, with Windows race coverage because the Pi race runtime cannot initialize. Decide explicitly whether to include the CLI follow-up when freezing the release; issue #72 remains partial either way. +The release candidate includes both formerly independent follow-ups: [server #160](https://github.com/MeshCore-Beacon/beacon-server/pull/160) for offline archive verification and [server #161](https://github.com/MeshCore-Beacon/beacon-server/pull/161) for ACK/TRACE/PING summaries. Their wider issues #72 and #99 remain partial. Integrity checks do not establish archive authenticity or restorability; packet references do not establish identity or delivery. -Independent follow-up [server #161](https://github.com/MeshCore-Beacon/beacon-server/pull/161), `0befdc5c`, adds ACK/TRACE/PING summaries in the existing packet display and also leaves the release queue independent. It merges cleanly with the reviewed stack. Combined server `5848d200` is on the Pi with web `42ba5fcb`; native PostgreSQL, Windows race, CI and public REST/live/browser checks pass. No schema/config changes or extra public admin access. Include it only if accepted when freezing; #99 remains partial for other requested formats. +Accepted server: `c02317a4ac7228d19cab498edfa1d61186c84626`. +Accepted web: `0f0a6ca51c7b2c3315db77954f61b30bbdeea5e2`. +The web source tree is identical to tested preview `42ba5fcb`; preserve that artifact's actual revision/source instead of relabeling it. The server differs from preview `5848d200` only in the verifier CLI/library/tests/docs; its verifier code and tests are identical to separately tested `262eae96`. The documentation additionally contains the accepted protected-download section. ## Review gates @@ -24,19 +26,27 @@ Independent follow-up [server #161](https://github.com/MeshCore-Beacon/beacon-se - [x] Refresh #55/#57 after the accepted #52/#53 squash. Their source trees were identical after the September 20 refresh; that history-only update needs no replacement Pi artifact. - [x] All eight application PRs pass their required checks on the published heads. Native PostgreSQL tests ran. The upstream web CodeQL job remains skipped under its existing policy and is not counted as a scan. -These checkmarks record completed corrections and validation, not maintainer approval. The four server PRs, four web PRs and docs #5 still await their merge/review decisions. - -Review order is server #149 -> #154 -> #157 -> #159. Web #55 waits for #157 to be merged **and deployed**; #57 waits for #159 to be merged and deployed and for #55. The icon fix can be reviewed independently. Traffic and Scopes have already landed together as web e01c090; do not replay #52. +All six server PRs and four web PRs have merged. Server #156/#158 and web #54/#56/#58/#60 are closed. Only docs #5 remains open. The helper retired accepted parents/overlays without rebasing or force-pushing any application branch. Future work starts from freshly fetched dev. Backend endpoints still need deployment before dependent pages on each operator's target; Pi evidence is not evidence of another deployment. ## Storage and retention boundary The September 17 drop-and-reset observation-partitioning design and implementation plan were explicitly superseded on September 19. They are historical reference only. Do not implement their table drop, history reset or process-local dedup replacement. -The stated replacement direction is lz4 compression, batched deletes, per-table autovacuum tuning and a seven-day default. Those changes are not present in the verified published server dev 951b79b (its example still says 30 days). Obtain and review the replacement contribution before describing it as shipped. Coordinate append-only migration numbers with that work; the old plan's proposed 035 is not evidence that a migration exists. +The stated replacement direction is lz4 compression, batched deletes, per-table autovacuum tuning and a seven-day default. Those changes are not present in the verified published server dev c02317a4 (its example still says 30 days). Obtain and review the replacement contribution before describing it as shipped. Coordinate append-only migration numbers with that work; the old plan's proposed 035 is not evidence that a migration exists. A consolidation release must document its actual retention behavior and capacity limits. A future production parity cutover also needs verified durable history coverage and recovery copies. Do not infer either from example configuration or the Pi's short history. -## Combined-candidate evidence +## Accepted-dev verification - 24 September + +- [x] Exact dev CI/image builds pass at server `c02317a4` and web `0f0a6ca5`; server CodeQL/coverage pass and web CodeQL remains skipped. +- [x] Server `c02317a4` built/tested natively on the Pi with real PostgreSQL: 1,194 passing test/subtest results. Signal, Paths, observer comparison, migration recovery, packet summaries and NULL observations ran. Two opt-in backup export/download integration suites were skipped; prior private restore/TLS checks remain separately dated evidence. +- [x] Accepted server is running on the preview. Source/asset hashes match; both feeds advance. Signal/Paths reconcile with SQL for global/regional 1/7/30-day selections, at 2-19 ms origin latency. Only three complete hours are populated; this does not prove 7/30-day history coverage. +- [x] Browser Signal/Paths charts and map load with LIVE status and no captured warnings/errors. Unchanged frontend assets keep their actual `42ba5fcb` build/source identity and prior 786-test evidence; accepted `0f0a6ca5` has the identical tree. +- [x] Only the Beacon app restarted; the other 22 containers and configuration/migration journal were preserved. Immediate rollback is server `5848d200` with unchanged web. The separate verifier retains its real `262eae96` binary/source identity. +- [x] All four review queues and overlays are empty. Start creates a branch from accepted dev with no dependency/rebase. Completed issues are closed; five broader issues remain open. +- [ ] Maintainer version selection, signed release commits, main promotion, tags and tag-built artifact verification. Stable releases remain v1.6.0 / v1.3.0. + +## Earlier combined-candidate evidence - 20 September - [x] September 20 unmodified Pi stability sample: 600 seconds / 41 samples, both feeds connected, 2,169 retained observations and no MQTT disconnect/deadline or HTTP 5xx. App/PostgreSQL CPU averaged 2.14%/3.96% of one core. This is a bounded health sample, not callback timing, #116 root-cause proof or a production-volume gate. [Result and limits](https://github.com/MeshCore-Beacon/beacon-server/issues/116#issuecomment-5753626821). - [x] Native Pi build/test of server `6be0f762` and web `42ba5fc`, including real PostgreSQL-to-HTTP checks and migration retry/concurrent refresh; 786 web tests pass. @@ -46,13 +56,12 @@ A consolidation release must document its actual retention behavior and capacity - [x] Real preview analytics reconcile with SQL for the materialized window. Browser charts, complete-hour text, small screens and error/empty/retry states are verified. Both MQTT feeds advance and the public browser reports LIVE. - [x] Current and rollback server/web artifacts, exact source offers and visible changelog match the running pair. Only the Beacon app restarted; the other 20 containers were preserved. Additive rollup migrations retain observations and are compatible with the previous binary. -The immediate server rollback from the packet-reference update is `6be0f762` with current web `42ba5fcb`. The previous frontend rollback `19672038` and earlier full pair, server `4f6679c8` / web `b1100972`, are also retained. The local deployment record preserves binaries, assets, source archives and runners. Rolling back the application keeps the additive rollup views; it does not remove history. +The September 20 packet-reference rollback was `6be0f762` with web `42ba5fcb`; it remains an older recovery point. The current immediate rollback is `5848d200` with the same frontend. The local deployment record preserves binaries, assets, source archives and runners. Application rollback keeps additive rollup views and does not remove history. ## Maintainer release handoff -1. Merge the reviewed queue in the declared order, using the shared refresh helper after accepted parents. Do not manually rebase every PR from scratch. -2. Deploy the accepted server before merging/deploying pages that need its new endpoints. Verify endpoint availability on the intended deployment, not just the Pi preview. -3. Freeze exact reviewed server/web dev heads and rerun required checks on them. Refresh the release notes with only changes actually included. -4. Follow the server contribution guide for a signed version/Swagger commit, dev-to-main fast-forward, tag and release CI. Web main has a prior release squash (`5ac36ce`) outside dev ancestry; maintainers must reconcile that stable history before choosing its promotion method. Do not silently overwrite main. -5. Verify the tag's Actions-built artifacts and matching source. Publish accurate release notes, upgrade/retention guidance, known gaps and rollback instructions. -6. After both releases, empty the accepted review queues and start the next feature directly from freshly fetched dev. Resume the parity roadmap; this milestone does not close partial #60/#72, #99, #116 or internationalization #12. +1. The application review queue is accepted. Freeze server `c02317a4` / web `0f0a6ca5`, or explicitly record any newer accepted changes before release. Confirm required checks on those exact heads. +2. Deploy the accepted server before dependent pages and verify both endpoints on the intended deployment. The Pi is a development validation target; production cutover remains with the owner. +3. Follow the server contribution guide for a signed version/Swagger commit, dev-to-main fast-forward, tag and release CI. Web main has the prior release squash `5ac36ce` outside dev ancestry; reconcile that stable history before promotion. Do not overwrite main. +4. Verify the tag's Actions-built artifacts and matching source. Publish accurate notes, upgrade/retention guidance, known gaps and rollback instructions. Versions/tags have not been chosen by this contribution. +5. Review the five remaining issues first when resuming development: server #60/#72/#99/#116 and web #12. These broad/partial issues remain open. After the consolidation release, Channel Activity is the next analytics page; this milestone does not establish full CoreScope parity. diff --git a/ROADMAP.md b/ROADMAP.md index 01d5c8c..9a70393 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,6 +1,6 @@ # Beacon parity and analytics roadmap -Updated 20 September 2026. This is the working roadmap for n30nex's ongoing contributions toward CoreScope feature parity. Maintainers decide acceptance and merge order; deployment owners handle the production switch. +Updated 24 September 2026. This is the working roadmap for n30nex's ongoing contributions toward CoreScope feature parity. Maintainers decide acceptance and merge order; deployment owners handle the production switch. Refresh GitHub issues, PR feedback and branch state before starting a phase. This document is a snapshot, and linked issues/PRs are the current source of truth. @@ -23,37 +23,28 @@ Current sites: | [beacon-docs](https://github.com/MeshCore-Beacon/beacon-docs) | Shared contracts, operator guidance and this roadmap | `main` | | [beacon-mobile](https://github.com/MeshCore-Beacon/beacon-mobile) | Mobile client; coordinate API compatibility | `main` | -## Current review queue +## Accepted consolidation batch -All active contribution PRs are available for review without draft status at the contributor's request. Dependencies still determine merge order. Review requests do not establish approval, and required checks must pass on the current published head. +All ten server/web contributions merged into `dev` on 24 September UTC. There are no open application PRs or remaining application rebase dependencies. [Docs #5](https://github.com/MeshCore-Beacon/beacon-docs/pull/5) is the only open contribution PR. -Server order: **#149 → #154 → #157 → #159**. +Accepted server is `c02317a4ac7228d19cab498edfa1d61186c84626`; accepted web is `0f0a6ca51c7b2c3315db77954f61b30bbdeea5e2`. CI and image builds pass at these heads; web CodeQL remains skipped and is not a security scan. Stable releases are still server **v1.6.0** and web **v1.3.0**. Merged dev is not a published stable release. -| PR | Scope | Issue disposition | +| Accepted PR | Scope | Merge commit | |---|---|---| -| [Server #149](https://github.com/MeshCore-Beacon/beacon-server/pull/149) | Protected operator-account lifecycle | Partial #60; these records are not browser logins | -| [Server #154](https://github.com/MeshCore-Beacon/beacon-server/pull/154) | Protected database/config backup download | Partial #72; login, validation/import and deployment-file coverage remain | -| [Server #157](https://github.com/MeshCore-Beacon/beacon-server/pull/157) | Bounded SNR/RSSI distributions and hourly statistics | Complete endpoint scope in #156 | -| [Server #159](https://github.com/MeshCore-Beacon/beacon-server/pull/159) | Bounded received-path and hash-width statistics | Complete endpoint scope in #158 | -| [Server #160](https://github.com/MeshCore-Beacon/beacon-server/pull/160) | Offline, non-destructive backup archive verification | Independent accepted-dev parent `951b79b`; partial #72, outside the shared stack | -| [Server #161](https://github.com/MeshCore-Beacon/beacon-server/pull/161) | ACK identifiers and TRACE/PING tags in packet summaries | Independent accepted-dev parent `951b79b`; partial #99, clean overlay on the shared stack | +| [Server #149](https://github.com/MeshCore-Beacon/beacon-server/pull/149) | add protected account lifecycle endpoints | `4d2642ab` | +| [Server #154](https://github.com/MeshCore-Beacon/beacon-server/pull/154) | add protected database and config download | `a25e2675` | +| [Server #157](https://github.com/MeshCore-Beacon/beacon-server/pull/157) | add bounded reception signal analytics | `6107578c` | +| [Server #159](https://github.com/MeshCore-Beacon/beacon-server/pull/159) | add bounded path and hash-width analytics | `8a3fa7e6` | +| [Server #160](https://github.com/MeshCore-Beacon/beacon-server/pull/160) | verify native archives offline without extraction | `20691dc5` | +| [Server #161](https://github.com/MeshCore-Beacon/beacon-server/pull/161) | summarize ACK and trace references | `c02317a4` | +| [Web #55](https://github.com/MeshCore-Beacon/beacon-web/pull/55) | add RF and signal analytics | `2405e1ac` | +| [Web #57](https://github.com/MeshCore-Beacon/beacon-web/pull/57) | add Paths and Hashes analytics | `929ba83c` | +| [Web #59](https://github.com/MeshCore-Beacon/beacon-web/pull/59) | distinguish analytics navigation icons | `c2ab29a5` | +| [Web #61](https://github.com/MeshCore-Beacon/beacon-web/pull/61) | omit reset and invalid node locations | `0f0a6ca5` | -#160 is an independent CLI follow-up and does not block the consolidation release. Candidate `262eae96` passes Windows/native Pi build, format, vet and unit checks, Windows race checks, malformed-input/fuzz cases, and exact compiled PostgreSQL export/verify/restore; CI/CodeQL pass. The Pi race runtime cannot initialize on its kernel and is recorded as unavailable. The CLI remains installed separately, with its own source link on the preview changelog. The validator checks archive structure and integrity, not authenticity, safe SQL or restorability. +Server #156/#158 and web #54/#56/#58/#60 are closed. Server #60/#72/#99/#116 and web #12 remain open: the accepted work completes slices, not their remaining scope. Archive verification establishes structure and integrity, not authenticity, SQL safety or restorability. Packet-carried ACK/TRACE/PING references are not identity or delivery guarantees. -#161 at `0befdc5c` also remains independent. Native builds of the standalone and combined candidates pass, including 28 PostgreSQL fixtures that keep each global/regional/backfill read to one query; Windows ingest race checks and CI/CodeQL pass. Public historical summaries and a live PING match packet detail, with 1280px/390px browser checks and no horizontal overflow. The existing web renders the new text without a client release. References are packet-carried checksums/tags, not identities or delivery/authentication guarantees. No message body or trace auth code is added. This follow-up does not block the consolidation release. - -Web order: **#59 → #55 → #57**, with independent map correction **#61** also ready. The icon correction #59 can land independently; #55/#57 wait until their server endpoints are merged and deployed. Traffic #52 landed verbatim in the #53 squash (`e01c090`); #52 was closed as included, and #53 is merged. Issues #50/#51 are closed. Observer comparison [#48](https://github.com/MeshCore-Beacon/beacon-web/pull/48) and foreign-node display [#49](https://github.com/MeshCore-Beacon/beacon-web/pull/49) are also merged. - -| PR | Scope | Dependency / issue | -|---|---|---| -| [Web #52](https://github.com/MeshCore-Beacon/beacon-web/pull/52) | Traffic heatmap, hourly trends and reception share | Included in merged #53; #50 closed | -| [Web #53](https://github.com/MeshCore-Beacon/beacon-web/pull/53) | Regional scope charts and exact counts | Merged as e01c090; #51 closed | -| [Web #61](https://github.com/MeshCore-Beacon/beacon-web/pull/61) | Omit reset/invalid map locations and false neighbour/path lines | Independent correction; #60 | -| [Web #59](https://github.com/MeshCore-Beacon/beacon-web/pull/59) | Distinct Traffic, Scopes and Compare icons | Independent correction; #58 | -| [Web #55](https://github.com/MeshCore-Beacon/beacon-web/pull/55) | RF / Signal charts and sample availability | After #59 and server #157 is merged and deployed; #54 | -| [Web #57](https://github.com/MeshCore-Beacon/beacon-web/pull/57) | Paths & Hashes charts and classification coverage | After #55 and server #159 is merged and deployed; #56 | - -The router foundation [server #155](https://github.com/MeshCore-Beacon/beacon-server/pull/155) is merged. The refresh workflow drops accepted squash parents, preserves focused feature deltas and handles changed branch history in isolated worktrees. See [the executable contributor workflow](CONTRIBUTOR_WORKFLOW.md). +The local workflow has removed the accepted entries and overlays without rebasing or force-pushing any application branch. Future contributions start directly from fresh `dev`. The unchanged web source tree matches the existing tested preview exactly; preserve its real build identity rather than labeling an older binary as a new build. See [the contributor workflow](CONTRIBUTOR_WORKFLOW.md) and [consolidation checklist](RELEASE-CHECKLIST.md). ## Delivered foundations @@ -75,18 +66,18 @@ The path page counts stored receptions, not unique devices. Flood paths accumula The September 20 review correction moves both new aggregate APIs onto materialized hourly snapshots, preserving reception/region semantics and normalizing polling windows to UTC hours. In a rolled-back million-row Pi fixture, Signal request queries took 1.8–77.5 ms and Paths 3.5–141.9 ms across custom/generic plans. Initial population took 14.4/8.4 seconds, refresh 16.8/6.2 seconds, and view/index storage was 6.5/11.3 MB respectively. These measurements cover the fixture, not the full production workload. See the [release consolidation checklist](RELEASE-CHECKLIST.md) for remaining gates. -The current combined preview has passed native Go/PostgreSQL/HTTP validation and **786 web tests**. Private backup checks cover version compatibility, feature-only startup failure, TLS/password files, cancellation and schema/data/sequence restoration. The review update passed 390/1280px browser checks, with ten distinct glyphs and explicit complete-hour text; earlier chart validation also covered 320/768px. Public Signal/Paths counts reconcile with SQL, both MQTT feeds advance and the browser reports LIVE. Current revisions and corresponding-source archives are on the preview's changelog page. +September 20 validation covered native Go/PostgreSQL/HTTP behavior and **786 web tests**, plus private backup compatibility, feature-only startup failure, TLS/password files, cancellation and schema/data/sequence restoration. That review update passed 390/1280px browser checks, with ten distinct glyphs and complete-hour text; earlier chart checks also covered 320/768px. This is historical evidence for the unchanged feature code. Current revisions and corresponding-source archives are on the preview's changelog page. -The current Pi build is server `5848d200` with unchanged web `42ba5fcb`. It composes #161 with the reviewed server stack; no existing PR branch was rewritten. Both MQTT feeds advance, schema/config remain unchanged and the other 20 containers were preserved. Packet-reference reads took 4.39–22.83 ms at the origin. The map correction and separate backup-verifier tool remain included. Current/rollback source and binaries are retained; only verified inactive staging was retired. +The Pi runs accepted server `c02317a4` with actual web build `42ba5fcb`, whose tree exactly equals accepted web `0f0a6ca5`. The server passed native Go build/format/vet/tests, with 1,194 passing test/subtest results and real PostgreSQL analytics, packet and migration checks. Two opt-in backup export/download integration suites were skipped; prior private restore/TLS evidence is separately dated. Both feeds advance, live Signal/Paths counts reconcile with SQL, public source/asset hashes match, and browser charts/map show LIVE without captured warnings/errors. The other 22 service containers were preserved. Immediate rollback is server `5848d200` with the same frontend; the separate tested archive verifier remains installed. Only three retained complete hours are populated, so 7/30-day selections do not establish durable history or production capacity. ## Next phases The September 20 #116 investigation has a new [current-build result](https://github.com/MeshCore-Beacon/beacon-server/issues/116#issuecomment-5753626821): a 600-second unmodified Pi capture kept both feeds connected and retained 2,169 new observations, with no ping timeout, disconnect, deadline, SQLSTATE error or HTTP 5xx response. App/PostgreSQL CPU averaged 2.14%/3.96% of one core. The preceding 3h39 log likewise has no MQTT loss or deadline error. Timestamp warnings were classified separately. This did not measure callback or pool-acquisition duration and does not establish the original cause or production capacity. No application, ordering, acknowledgement or service change was made; #116 remains open. Further capture should follow a recurrence or meaningful workload change, rather than repeatedly sampling the same healthy state. -1. **Finish the current queue and prepare a consolidation release.** Address all four server reviews and the analytics icon regression [web #58](https://github.com/MeshCore-Beacon/beacon-web/issues/58). Validate the combined candidate on the Pi, retain backend-before-frontend deployment order, then hand reviewed `dev` candidates to maintainers for the server/web main releases. Follow [RELEASE-CHECKLIST.md](RELEASE-CHECKLIST.md). Pause new analytics until this breakpoint; this release does not claim complete CoreScope parity. +1. **Publish the coordinated consolidation releases.** Accepted-dev native/Pi/public validation and the changelog/source handoff are complete. Maintainers choose versions, create signed release commits/tags, promote main and verify release artifacts. Follow [RELEASE-CHECKLIST.md](RELEASE-CHECKLIST.md). Pause new analytics until this breakpoint; this release does not claim complete CoreScope parity. 2. **After the consolidation release: Channel Activity analytics.** Inspect existing aggregates and add bounded channel traffic/trend views. Define messages versus receptions, unknown/encrypted-channel coverage and the behavior when keys are unavailable. Keep keys and message contents out of aggregate responses; avoid raw-message paging to build statistics. 3. **Observed ambiguity and topology.** Separate static prefix conflicts from observed unresolved/ambiguous paths. Add justified route-pattern, neighbour, hop and distance views with clear provenance and bounded work. -4. **Administration and backup slices.** Non-destructive archive validation is ready in independent #160. Resolve browser login/session and restore authorization before an import endpoint; keep deployment-file coverage and remote/scheduled backup separate. Test restoration against disposable databases. These follow-ups do not delay the current release. +4. **Administration and backup slices.** Non-destructive archive validation is merged in #160. Resolve browser login/session and restore authorization before an import endpoint; keep deployment-file coverage and remote/scheduled backup separate. Test restoration against disposable databases. These follow-ups do not delay the current release. 5. **Production evidence and handoff.** Reconcile history, operational limits and the parity matrix below before preparing a release/cutover handoff. ## Listed work still open @@ -94,14 +85,12 @@ The September 20 #116 investigation has a new [current-build result](https://git | Issue | Remaining scope | |---|---| | [Server #116](https://github.com/MeshCore-Beacon/beacon-server/issues/116) | No recurrence in the September 20 retained log / ten-minute capture; still needs an attributable event with callback/pool timing | -| [Server #99](https://github.com/MeshCore-Beacon/beacon-server/issues/99) | Advert names are accepted; ACK/TRACE/PING references in #161 await review; define any remaining packet-type formats | +| [Server #99](https://github.com/MeshCore-Beacon/beacon-server/issues/99) | Advert names and ACK/TRACE/PING references are accepted; define any remaining packet-type formats | | [Server #60](https://github.com/MeshCore-Beacon/beacon-server/issues/60) | Remaining administration/worker/persistence behavior; account records do not establish login sessions | -| [Server #72](https://github.com/MeshCore-Beacon/beacon-server/issues/72) | Archive validation #160 awaits review; import, browser access, deployment-file coverage and remote/scheduled backup remain | +| [Server #72](https://github.com/MeshCore-Beacon/beacon-server/issues/72) | Download #154 and archive validation #160 are accepted; import, browser access, deployment-file coverage and remote/scheduled backup remain | | [Web #12](https://github.com/MeshCore-Beacon/beacon-web/issues/12) | Supported-language and formatting scope for internationalization | -| [Web #60](https://github.com/MeshCore-Beacon/beacon-web/issues/60) | Map location-reset correction #61 is deployed and awaiting merge | -| [Web #58](https://github.com/MeshCore-Beacon/beacon-web/issues/58) | Distinct analytics navigation icons, including pending RF/Signal and Paths pages | -The analytics endpoint/page issues remain open while their corresponding PRs await merge. Use closing references only when a PR completes the issue's accepted scope; use related references for partial work. +The six completed analytics/icon/map issues are closed. Refresh the five remaining issues first at every continuation; accepted partial contributions are not grounds to close broader issues. Use closing references only when a PR completes the issue's accepted scope; use related references for partial work. ## Production parity matrix From 289a618fcd20817449f3924bfc62874748556362 Mon Sep 17 00:00:00 2001 From: n30nex Date: Thu, 24 Sep 2026 06:19:47 -0400 Subject: [PATCH 11/69] docs(roadmap): record French navigation delivery and remaining scope --- RELEASE-CHECKLIST.md | 10 ++++++---- ROADMAP.md | 25 +++++++++++++++++-------- 2 files changed, 23 insertions(+), 12 deletions(-) diff --git a/RELEASE-CHECKLIST.md b/RELEASE-CHECKLIST.md index 1bd9a86..79ca04c 100644 --- a/RELEASE-CHECKLIST.md +++ b/RELEASE-CHECKLIST.md @@ -1,6 +1,6 @@ # Server/web consolidation release -Status: accepted-dev validation and release handoff, 24 September 2026. All ten application/CLI PRs are merged. Stable tags remain unchanged; this is not a published release or a full CoreScope parity claim. +Status: accepted-dev validation and release handoff, 24 September 2026. All ten application/CLI PRs in the consolidation batch are merged. Stable tags remain unchanged; this is not a published release or a full CoreScope parity claim. ## Scope and stop point @@ -26,7 +26,7 @@ The web source tree is identical to tested preview `42ba5fcb`; preserve that art - [x] Refresh #55/#57 after the accepted #52/#53 squash. Their source trees were identical after the September 20 refresh; that history-only update needs no replacement Pi artifact. - [x] All eight application PRs pass their required checks on the published heads. Native PostgreSQL tests ran. The upstream web CodeQL job remains skipped under its existing policy and is not counted as a scan. -All six server PRs and four web PRs have merged. Server #156/#158 and web #54/#56/#58/#60 are closed. Only docs #5 remains open. The helper retired accepted parents/overlays without rebasing or force-pushing any application branch. Future work starts from freshly fetched dev. Backend endpoints still need deployment before dependent pages on each operator's target; Pi evidence is not evidence of another deployment. +All six server PRs and four web PRs have merged. Server #156/#158 and web #54/#56/#58/#60 are closed. Docs #5 remains open; new independent web #62 is outside this frozen batch. The helper retired accepted parents/overlays without rebasing or force-pushing any application branch. Future work starts from freshly fetched dev. Backend endpoints still need deployment before dependent pages on each operator's target; Pi evidence is not evidence of another deployment. ## Storage and retention boundary @@ -36,6 +36,8 @@ The stated replacement direction is lz4 compression, batched deletes, per-table A consolidation release must document its actual retention behavior and capacity limits. A future production parity cutover also needs verified durable history coverage and recovery copies. Do not infer either from example configuration or the Pi's short history. +The live preview now additionally includes independent language-support PR #62 at web `b8d4dbf8`. The tested accepted frontend `42ba5fcb` (tree-equivalent to accepted `0f0a6ca5`) remains the immediate rollback. The release freeze below still describes the accepted batch; including #62 requires a separate maintainer decision. Its native 799-test/public browser evidence is recorded in the roadmap. + ## Accepted-dev verification - 24 September - [x] Exact dev CI/image builds pass at server `c02317a4` and web `0f0a6ca5`; server CodeQL/coverage pass and web CodeQL remains skipped. @@ -43,7 +45,7 @@ A consolidation release must document its actual retention behavior and capacity - [x] Accepted server is running on the preview. Source/asset hashes match; both feeds advance. Signal/Paths reconcile with SQL for global/regional 1/7/30-day selections, at 2-19 ms origin latency. Only three complete hours are populated; this does not prove 7/30-day history coverage. - [x] Browser Signal/Paths charts and map load with LIVE status and no captured warnings/errors. Unchanged frontend assets keep their actual `42ba5fcb` build/source identity and prior 786-test evidence; accepted `0f0a6ca5` has the identical tree. - [x] Only the Beacon app restarted; the other 22 containers and configuration/migration journal were preserved. Immediate rollback is server `5848d200` with unchanged web. The separate verifier retains its real `262eae96` binary/source identity. -- [x] All four review queues and overlays are empty. Start creates a branch from accepted dev with no dependency/rebase. Completed issues are closed; five broader issues remain open. +- [x] Accepted review queues/overlays were retired. Start created the independent language branch directly from accepted dev; #62 is now the only active web entry. Completed issues are closed; five broader issues remain open. - [ ] Maintainer version selection, signed release commits, main promotion, tags and tag-built artifact verification. Stable releases remain v1.6.0 / v1.3.0. ## Earlier combined-candidate evidence - 20 September @@ -56,7 +58,7 @@ A consolidation release must document its actual retention behavior and capacity - [x] Real preview analytics reconcile with SQL for the materialized window. Browser charts, complete-hour text, small screens and error/empty/retry states are verified. Both MQTT feeds advance and the public browser reports LIVE. - [x] Current and rollback server/web artifacts, exact source offers and visible changelog match the running pair. Only the Beacon app restarted; the other 20 containers were preserved. Additive rollup migrations retain observations and are compatible with the previous binary. -The September 20 packet-reference rollback was `6be0f762` with web `42ba5fcb`; it remains an older recovery point. The current immediate rollback is `5848d200` with the same frontend. The local deployment record preserves binaries, assets, source archives and runners. Application rollback keeps additive rollup views and does not remove history. +The current immediate rollback restores frontend `42ba5fcb` with server `c02317a4`. Restore that frontend before using the older consolidation server rollback to `5848d200`; its metadata describes the accepted frontend. The September 20 packet-reference rollback to `6be0f762` is an older recovery point. Exact artifacts/runners are retained; application rollback keeps additive rollup views and does not remove history. ## Maintainer release handoff diff --git a/ROADMAP.md b/ROADMAP.md index 9a70393..6f08733 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -25,7 +25,7 @@ Current sites: ## Accepted consolidation batch -All ten server/web contributions merged into `dev` on 24 September UTC. There are no open application PRs or remaining application rebase dependencies. [Docs #5](https://github.com/MeshCore-Beacon/beacon-docs/pull/5) is the only open contribution PR. +All ten server/web contributions merged into `dev` on 24 September UTC. The accepted batch has no remaining rebase dependencies. Independent [web #62](https://github.com/MeshCore-Beacon/beacon-web/pull/62) now addresses language-support issue #12; [docs #5](https://github.com/MeshCore-Beacon/beacon-docs/pull/5) also remains open. Accepted server is `c02317a4ac7228d19cab498edfa1d61186c84626`; accepted web is `0f0a6ca51c7b2c3315db77954f61b30bbdeea5e2`. CI and image builds pass at these heads; web CodeQL remains skipped and is not a security scan. Stable releases are still server **v1.6.0** and web **v1.3.0**. Merged dev is not a published stable release. @@ -44,7 +44,15 @@ Accepted server is `c02317a4ac7228d19cab498edfa1d61186c84626`; accepted web is ` Server #156/#158 and web #54/#56/#58/#60 are closed. Server #60/#72/#99/#116 and web #12 remain open: the accepted work completes slices, not their remaining scope. Archive verification establishes structure and integrity, not authenticity, SQL safety or restorability. Packet-carried ACK/TRACE/PING references are not identity or delivery guarantees. -The local workflow has removed the accepted entries and overlays without rebasing or force-pushing any application branch. Future contributions start directly from fresh `dev`. The unchanged web source tree matches the existing tested preview exactly; preserve its real build identity rather than labeling an older binary as a new build. See [the contributor workflow](CONTRIBUTOR_WORKFLOW.md) and [consolidation checklist](RELEASE-CHECKLIST.md). +The workflow removed the accepted entries and overlays without rebasing or force-pushing any application branch. The language foundation started directly from fresh `dev` and is the sole active web entry. Overlapping follow-ups use its declared parent through the helper. The unchanged web source tree matches the existing tested preview exactly; preserve its real build identity rather than labeling an older binary as a new build. See [the contributor workflow](CONTRIBUTOR_WORKFLOW.md) and [consolidation checklist](RELEASE-CHECKLIST.md). + +## Current independent issue work + +[Web #62](https://github.com/MeshCore-Beacon/beacon-web/pull/62), `b8d4dbf80627411e17875f9a1e11b220db94ffe7`, is the first slice of [language support #12](https://github.com/MeshCore-Beacon/beacon-web/issues/12). English remains the default; French covers navigation, region/theme controls, connection status and lazy-page loading. A native language picker saves the browser choice. Bundled catalogs are discovered automatically and missing/empty translations fall back to English. Contributor guidance accompanies the feature. + +The exact candidate passes build/lint and all 799 tests on Windows and the Pi. Public browser checks show French stays LIVE, survives reloads, preserves canonical links and fits a 320px viewport. Source/asset hashes match; the initial JS adds about 16.1 KiB gzip. All 23 service containers and the accepted server were preserved; no application rebase was needed. CI passes; web CodeQL remains skipped. Detailed pages, chart/dialog text and localized number/time formatting remain follow-ups, so #12 stays open. Physical Safari remains unverified. + +This independent PR does not expand the frozen consolidation release. The Pi runs server `c02317a4` / web `b8d4dbf8`; immediate frontend rollback is tested `42ba5fcb`, identical in source to accepted web `0f0a6ca5`. The [preview changelog/source](https://canadaverse.org/beacon-dev/source.html) identifies both the accepted batch and the additional language candidate. ## Delivered foundations @@ -68,17 +76,18 @@ The September 20 review correction moves both new aggregate APIs onto materializ September 20 validation covered native Go/PostgreSQL/HTTP behavior and **786 web tests**, plus private backup compatibility, feature-only startup failure, TLS/password files, cancellation and schema/data/sequence restoration. That review update passed 390/1280px browser checks, with ten distinct glyphs and complete-hour text; earlier chart checks also covered 320/768px. This is historical evidence for the unchanged feature code. Current revisions and corresponding-source archives are on the preview's changelog page. -The Pi runs accepted server `c02317a4` with actual web build `42ba5fcb`, whose tree exactly equals accepted web `0f0a6ca5`. The server passed native Go build/format/vet/tests, with 1,194 passing test/subtest results and real PostgreSQL analytics, packet and migration checks. Two opt-in backup export/download integration suites were skipped; prior private restore/TLS evidence is separately dated. Both feeds advance, live Signal/Paths counts reconcile with SQL, public source/asset hashes match, and browser charts/map show LIVE without captured warnings/errors. The other 22 service containers were preserved. Immediate rollback is server `5848d200` with the same frontend; the separate tested archive verifier remains installed. Only three retained complete hours are populated, so 7/30-day selections do not establish durable history or production capacity. +The September 24 consolidation check built accepted server `c02317a4` and retained web `42ba5fcb`, identical in source to accepted `0f0a6ca5`. Its native PostgreSQL and public/browser evidence remains in the release checklist. The current Pi frontend additionally includes #62 as documented above; the accepted frontend is retained for rollback. The three-hour retained analytics sample does not establish 7/30-day history or production capacity. ## Next phases The September 20 #116 investigation has a new [current-build result](https://github.com/MeshCore-Beacon/beacon-server/issues/116#issuecomment-5753626821): a 600-second unmodified Pi capture kept both feeds connected and retained 2,169 new observations, with no ping timeout, disconnect, deadline, SQLSTATE error or HTTP 5xx response. App/PostgreSQL CPU averaged 2.14%/3.96% of one core. The preceding 3h39 log likewise has no MQTT loss or deadline error. Timestamp warnings were classified separately. This did not measure callback or pool-acquisition duration and does not establish the original cause or production capacity. No application, ordering, acknowledgement or service change was made; #116 remains open. Further capture should follow a recurrence or meaningful workload change, rather than repeatedly sampling the same healthy state. 1. **Publish the coordinated consolidation releases.** Accepted-dev native/Pi/public validation and the changelog/source handoff are complete. Maintainers choose versions, create signed release commits/tags, promote main and verify release artifacts. Follow [RELEASE-CHECKLIST.md](RELEASE-CHECKLIST.md). Pause new analytics until this breakpoint; this release does not claim complete CoreScope parity. -2. **After the consolidation release: Channel Activity analytics.** Inspect existing aggregates and add bounded channel traffic/trend views. Define messages versus receptions, unknown/encrypted-channel coverage and the behavior when keys are unavailable. Keep keys and message contents out of aggregate responses; avoid raw-message paging to build statistics. -3. **Observed ambiguity and topology.** Separate static prefix conflicts from observed unresolved/ambiguous paths. Add justified route-pattern, neighbour, hop and distance views with clear provenance and bounded work. -4. **Administration and backup slices.** Non-destructive archive validation is merged in #160. Resolve browser login/session and restore authorization before an import endpoint; keep deployment-file coverage and remote/scheduled backup separate. Test restoration against disposable databases. These follow-ups do not delay the current release. -5. **Production evidence and handoff.** Reconcile history, operational limits and the parity matrix below before preparing a release/cutover handoff. +2. **Continue listed issue #12 in small screen groups.** The English/French foundation is ready in #62. Translate remaining page, chart and dialog text while preserving identifiers and measurement semantics; keep partial coverage explicit. Refresh all open issues/reviews before choosing the next group. +3. **After the consolidation release: Channel Activity analytics.** Inspect existing aggregates and add bounded channel traffic/trend views. Define messages versus receptions, unknown/encrypted-channel coverage and the behavior when keys are unavailable. Keep keys and message contents out of aggregate responses; avoid raw-message paging to build statistics. +4. **Observed ambiguity and topology.** Separate static prefix conflicts from observed unresolved/ambiguous paths. Add justified route-pattern, neighbour, hop and distance views with clear provenance and bounded work. +5. **Administration and backup slices.** Non-destructive archive validation is merged in #160. Resolve browser login/session and restore authorization before an import endpoint; keep deployment-file coverage and remote/scheduled backup separate. Test restoration against disposable databases. These follow-ups do not delay the current release. +6. **Production evidence and handoff.** Reconcile history, operational limits and the parity matrix below before preparing a release/cutover handoff. ## Listed work still open @@ -88,7 +97,7 @@ The September 20 #116 investigation has a new [current-build result](https://git | [Server #99](https://github.com/MeshCore-Beacon/beacon-server/issues/99) | Advert names and ACK/TRACE/PING references are accepted; define any remaining packet-type formats | | [Server #60](https://github.com/MeshCore-Beacon/beacon-server/issues/60) | Remaining administration/worker/persistence behavior; account records do not establish login sessions | | [Server #72](https://github.com/MeshCore-Beacon/beacon-server/issues/72) | Download #154 and archive validation #160 are accepted; import, browser access, deployment-file coverage and remote/scheduled backup remain | -| [Web #12](https://github.com/MeshCore-Beacon/beacon-web/issues/12) | Supported-language and formatting scope for internationalization | +| [Web #12](https://github.com/MeshCore-Beacon/beacon-web/issues/12) | English/French navigation foundation #62 is in review; detailed screens, chart/dialog text and formatting remain | The six completed analytics/icon/map issues are closed. Refresh the five remaining issues first at every continuation; accepted partial contributions are not grounds to close broader issues. Use closing references only when a PR completes the issue's accepted scope; use related references for partial work. From d8e4f67cebb54c206305fd81d8c896a69b0f29d1 Mon Sep 17 00:00:00 2001 From: n30nex Date: Thu, 24 Sep 2026 08:49:38 -0400 Subject: [PATCH 12/69] docs(roadmap): record matching language dropdown review fix --- RELEASE-CHECKLIST.md | 4 ++-- ROADMAP.md | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/RELEASE-CHECKLIST.md b/RELEASE-CHECKLIST.md index 79ca04c..47b0217 100644 --- a/RELEASE-CHECKLIST.md +++ b/RELEASE-CHECKLIST.md @@ -36,7 +36,7 @@ The stated replacement direction is lz4 compression, batched deletes, per-table A consolidation release must document its actual retention behavior and capacity limits. A future production parity cutover also needs verified durable history coverage and recovery copies. Do not infer either from example configuration or the Pi's short history. -The live preview now additionally includes independent language-support PR #62 at web `b8d4dbf8`. The tested accepted frontend `42ba5fcb` (tree-equivalent to accepted `0f0a6ca5`) remains the immediate rollback. The release freeze below still describes the accepted batch; including #62 requires a separate maintainer decision. Its native 799-test/public browser evidence is recorded in the roadmap. +The live preview now additionally includes independent language-support PR #62 at web `1c77f200`, including the header-dropdown review correction. The prior language build `b8d4dbf8` is the immediate rollback; accepted frontend `42ba5fcb` (tree-equivalent to accepted `0f0a6ca5`) is also retained. The release freeze below still describes the accepted batch; including #62 requires a separate maintainer decision. Its native 802-test/public browser evidence is recorded in the roadmap. ## Accepted-dev verification - 24 September @@ -58,7 +58,7 @@ The live preview now additionally includes independent language-support PR #62 a - [x] Real preview analytics reconcile with SQL for the materialized window. Browser charts, complete-hour text, small screens and error/empty/retry states are verified. Both MQTT feeds advance and the public browser reports LIVE. - [x] Current and rollback server/web artifacts, exact source offers and visible changelog match the running pair. Only the Beacon app restarted; the other 20 containers were preserved. Additive rollup migrations retain observations and are compatible with the previous binary. -The current immediate rollback restores frontend `42ba5fcb` with server `c02317a4`. Restore that frontend before using the older consolidation server rollback to `5848d200`; its metadata describes the accepted frontend. The September 20 packet-reference rollback to `6be0f762` is an older recovery point. Exact artifacts/runners are retained; application rollback keeps additive rollup views and does not remove history. +The current immediate rollback restores frontend `b8d4dbf8` with server `c02317a4`. Restore accepted frontend `42ba5fcb` before using the older consolidation server rollback to `5848d200`; its metadata describes the accepted frontend. The September 20 packet-reference rollback to `6be0f762` is an older recovery point. Exact artifacts/runners are retained; application rollback keeps additive rollup views and does not remove history. ## Maintainer release handoff diff --git a/ROADMAP.md b/ROADMAP.md index 6f08733..78581dc 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -48,11 +48,11 @@ The workflow removed the accepted entries and overlays without rebasing or force ## Current independent issue work -[Web #62](https://github.com/MeshCore-Beacon/beacon-web/pull/62), `b8d4dbf80627411e17875f9a1e11b220db94ffe7`, is the first slice of [language support #12](https://github.com/MeshCore-Beacon/beacon-web/issues/12). English remains the default; French covers navigation, region/theme controls, connection status and lazy-page loading. A native language picker saves the browser choice. Bundled catalogs are discovered automatically and missing/empty translations fall back to English. Contributor guidance accompanies the feature. +[Web #62](https://github.com/MeshCore-Beacon/beacon-web/pull/62), `1c77f200e583908a5b26b369f500410a1d945d15`, is the first slice of [language support #12](https://github.com/MeshCore-Beacon/beacon-web/issues/12). English remains the default; French covers navigation, region/theme controls, connection status and lazy-page loading. The language picker now matches the region dropdown: compact label/code/arrow, 26px height, 12px monospace text and shared panel styling. Keyboard selection and Escape return focus to the trigger. The browser choice is saved. Bundled catalogs are discovered automatically and missing/empty translations fall back to English. Contributor guidance accompanies the feature. -The exact candidate passes build/lint and all 799 tests on Windows and the Pi. Public browser checks show French stays LIVE, survives reloads, preserves canonical links and fits a 320px viewport. Source/asset hashes match; the initial JS adds about 16.1 KiB gzip. All 23 service containers and the accepted server were preserved; no application rebase was needed. CI passes; web CodeQL remains skipped. Detailed pages, chart/dialog text and localized number/time formatting remain follow-ups, so #12 stays open. Physical Safari remains unverified. +The exact candidate passes build/lint and all 802 tests on Windows and the Pi. Public browser checks show French stays LIVE, survives reloads, preserves canonical links and fits a 320px viewport. Source/asset hashes match; the initial JS adds about 16.3 KiB gzip. All 23 service containers and the accepted server were preserved; no application rebase was needed. CI passes; web CodeQL remains skipped. Detailed pages, chart/dialog text and localized number/time formatting remain follow-ups, so #12 stays open. Physical Safari remains unverified. -This independent PR does not expand the frozen consolidation release. The Pi runs server `c02317a4` / web `b8d4dbf8`; immediate frontend rollback is tested `42ba5fcb`, identical in source to accepted web `0f0a6ca5`. The [preview changelog/source](https://canadaverse.org/beacon-dev/source.html) identifies both the accepted batch and the additional language candidate. +This independent PR does not expand the frozen consolidation release. The Pi runs server `c02317a4` / web `1c77f200`; immediate frontend rollback is `b8d4dbf8`. Accepted frontend `42ba5fcb`, identical in source to accepted web `0f0a6ca5`, remains an older recovery point. The [preview changelog/source](https://canadaverse.org/beacon-dev/source.html) identifies both the accepted batch and the additional language candidate. ## Delivered foundations From 0fce515cab3323bceb25de36a6e656e60f086731 Mon Sep 17 00:00:00 2001 From: n30nex Date: Thu, 24 Sep 2026 10:30:13 -0400 Subject: [PATCH 13/69] docs(roadmap): record French signal analytics delivery --- RELEASE-CHECKLIST.md | 8 ++++---- ROADMAP.md | 20 +++++++++++--------- 2 files changed, 15 insertions(+), 13 deletions(-) diff --git a/RELEASE-CHECKLIST.md b/RELEASE-CHECKLIST.md index 47b0217..c46ed86 100644 --- a/RELEASE-CHECKLIST.md +++ b/RELEASE-CHECKLIST.md @@ -26,7 +26,7 @@ The web source tree is identical to tested preview `42ba5fcb`; preserve that art - [x] Refresh #55/#57 after the accepted #52/#53 squash. Their source trees were identical after the September 20 refresh; that history-only update needs no replacement Pi artifact. - [x] All eight application PRs pass their required checks on the published heads. Native PostgreSQL tests ran. The upstream web CodeQL job remains skipped under its existing policy and is not counted as a scan. -All six server PRs and four web PRs have merged. Server #156/#158 and web #54/#56/#58/#60 are closed. Docs #5 remains open; new independent web #62 is outside this frozen batch. The helper retired accepted parents/overlays without rebasing or force-pushing any application branch. Future work starts from freshly fetched dev. Backend endpoints still need deployment before dependent pages on each operator's target; Pi evidence is not evidence of another deployment. +All six server PRs and four web PRs have merged. Server #156/#158 and web #54/#56/#58/#60 are closed. Docs #5 remains open; language foundation #62 has since merged, and new translation #63 is outside this original frozen batch. The helper retired accepted parents/overlays without rebasing or force-pushing any application branch. Future work starts from freshly fetched dev. Backend endpoints still need deployment before dependent pages on each operator's target; Pi evidence is not evidence of another deployment. ## Storage and retention boundary @@ -36,7 +36,7 @@ The stated replacement direction is lz4 compression, batched deletes, per-table A consolidation release must document its actual retention behavior and capacity limits. A future production parity cutover also needs verified durable history coverage and recovery copies. Do not infer either from example configuration or the Pi's short history. -The live preview now additionally includes independent language-support PR #62 at web `1c77f200`, including the header-dropdown review correction. The prior language build `b8d4dbf8` is the immediate rollback; accepted frontend `42ba5fcb` (tree-equivalent to accepted `0f0a6ca5`) is also retained. The release freeze below still describes the accepted batch; including #62 requires a separate maintainer decision. Its native 802-test/public browser evidence is recorded in the roadmap. +Language foundation #62 has since merged as accepted web dev `6d3edbb6`, source-equivalent to tested build `1c77f200`. The live preview now additionally includes RF / Signal translation #63 at `01c6d1aa`, with 812 passing native/Windows tests and public browser/source verification. Immediate frontend rollback is `1c77f200`. The original release freeze below remains separately recorded; maintainers choose whether to advance it to include these later contributions. ## Accepted-dev verification - 24 September @@ -45,7 +45,7 @@ The live preview now additionally includes independent language-support PR #62 a - [x] Accepted server is running on the preview. Source/asset hashes match; both feeds advance. Signal/Paths reconcile with SQL for global/regional 1/7/30-day selections, at 2-19 ms origin latency. Only three complete hours are populated; this does not prove 7/30-day history coverage. - [x] Browser Signal/Paths charts and map load with LIVE status and no captured warnings/errors. Unchanged frontend assets keep their actual `42ba5fcb` build/source identity and prior 786-test evidence; accepted `0f0a6ca5` has the identical tree. - [x] Only the Beacon app restarted; the other 22 containers and configuration/migration journal were preserved. Immediate rollback is server `5848d200` with unchanged web. The separate verifier retains its real `262eae96` binary/source identity. -- [x] Accepted review queues/overlays were retired. Start created the independent language branch directly from accepted dev; #62 is now the only active web entry. Completed issues are closed; five broader issues remain open. +- [x] Accepted review queues/overlays were retired. Start created the independent language branch directly from accepted dev; #63 is now the only active web entry. Completed issues are closed; five broader issues remain open. - [ ] Maintainer version selection, signed release commits, main promotion, tags and tag-built artifact verification. Stable releases remain v1.6.0 / v1.3.0. ## Earlier combined-candidate evidence - 20 September @@ -58,7 +58,7 @@ The live preview now additionally includes independent language-support PR #62 a - [x] Real preview analytics reconcile with SQL for the materialized window. Browser charts, complete-hour text, small screens and error/empty/retry states are verified. Both MQTT feeds advance and the public browser reports LIVE. - [x] Current and rollback server/web artifacts, exact source offers and visible changelog match the running pair. Only the Beacon app restarted; the other 20 containers were preserved. Additive rollup migrations retain observations and are compatible with the previous binary. -The current immediate rollback restores frontend `b8d4dbf8` with server `c02317a4`. Restore accepted frontend `42ba5fcb` before using the older consolidation server rollback to `5848d200`; its metadata describes the accepted frontend. The September 20 packet-reference rollback to `6be0f762` is an older recovery point. Exact artifacts/runners are retained; application rollback keeps additive rollup views and does not remove history. +The current immediate rollback restores frontend `1c77f200` with server `c02317a4`. Restore accepted frontend `42ba5fcb` before using the older consolidation server rollback to `5848d200`; its metadata describes the accepted frontend. The September 20 packet-reference rollback to `6be0f762` is an older recovery point. Exact artifacts/runners are retained; application rollback keeps additive rollup views and does not remove history. ## Maintainer release handoff diff --git a/ROADMAP.md b/ROADMAP.md index 78581dc..c0a7843 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -25,9 +25,9 @@ Current sites: ## Accepted consolidation batch -All ten server/web contributions merged into `dev` on 24 September UTC. The accepted batch has no remaining rebase dependencies. Independent [web #62](https://github.com/MeshCore-Beacon/beacon-web/pull/62) now addresses language-support issue #12; [docs #5](https://github.com/MeshCore-Beacon/beacon-docs/pull/5) also remains open. +All ten server/web contributions merged into `dev` on 24 September UTC. The accepted batch has no remaining rebase dependencies. Language foundation [web #62](https://github.com/MeshCore-Beacon/beacon-web/pull/62) has also merged as `6d3edbb6`. Only the new Signal translation [web #63](https://github.com/MeshCore-Beacon/beacon-web/pull/63) and [docs #5](https://github.com/MeshCore-Beacon/beacon-docs/pull/5) remain open. -Accepted server is `c02317a4ac7228d19cab498edfa1d61186c84626`; accepted web is `0f0a6ca51c7b2c3315db77954f61b30bbdeea5e2`. CI and image builds pass at these heads; web CodeQL remains skipped and is not a security scan. Stable releases are still server **v1.6.0** and web **v1.3.0**. Merged dev is not a published stable release. +The original consolidation freeze is server `c02317a4ac7228d19cab498edfa1d61186c84626` / web `0f0a6ca51c7b2c3315db77954f61b30bbdeea5e2`. CI and image builds pass at those heads; web CodeQL remains skipped and is not a security scan. Current web dev has since advanced to `6d3edbb6` through #62. Stable releases are still server **v1.6.0** and web **v1.3.0**. Merged dev is not a published stable release. | Accepted PR | Scope | Merge commit | |---|---|---| @@ -44,15 +44,17 @@ Accepted server is `c02317a4ac7228d19cab498edfa1d61186c84626`; accepted web is ` Server #156/#158 and web #54/#56/#58/#60 are closed. Server #60/#72/#99/#116 and web #12 remain open: the accepted work completes slices, not their remaining scope. Archive verification establishes structure and integrity, not authenticity, SQL safety or restorability. Packet-carried ACK/TRACE/PING references are not identity or delivery guarantees. -The workflow removed the accepted entries and overlays without rebasing or force-pushing any application branch. The language foundation started directly from fresh `dev` and is the sole active web entry. Overlapping follow-ups use its declared parent through the helper. The unchanged web source tree matches the existing tested preview exactly; preserve its real build identity rather than labeling an older binary as a new build. See [the contributor workflow](CONTRIBUTOR_WORKFLOW.md) and [consolidation checklist](RELEASE-CHECKLIST.md). +The workflow removed the accepted entries and overlays without rebasing or force-pushing any application branch. Signal translation #63 started directly from fresh dev `6d3edbb6` and is the sole active web entry; future overlapping follow-ups use that declared parent through the helper. Preserve actual build identities when accepted commits have identical source trees. See [the contributor workflow](CONTRIBUTOR_WORKFLOW.md) and [consolidation checklist](RELEASE-CHECKLIST.md). ## Current independent issue work -[Web #62](https://github.com/MeshCore-Beacon/beacon-web/pull/62), `1c77f200e583908a5b26b369f500410a1d945d15`, is the first slice of [language support #12](https://github.com/MeshCore-Beacon/beacon-web/issues/12). English remains the default; French covers navigation, region/theme controls, connection status and lazy-page loading. The language picker now matches the region dropdown: compact label/code/arrow, 26px height, 12px monospace text and shared panel styling. Keyboard selection and Escape return focus to the trigger. The browser choice is saved. Bundled catalogs are discovered automatically and missing/empty translations fall back to English. Contributor guidance accompanies the feature. +Language foundation #62 is merged as accepted dev `6d3edbb61bb79fe7aa336c5ae93a6c0a3f6f7545`. Its accepted tree equals the tested `1c77f200` build, so the merge needed no rebase, artifact relabel or Pi rebuild. -The exact candidate passes build/lint and all 802 tests on Windows and the Pi. Public browser checks show French stays LIVE, survives reloads, preserves canonical links and fits a 320px viewport. Source/asset hashes match; the initial JS adds about 16.3 KiB gzip. All 23 service containers and the accepted server were preserved; no application rebase was needed. CI passes; web CodeQL remains skipped. Detailed pages, chart/dialog text and localized number/time formatting remain follow-ups, so #12 stays open. Physical Safari remains unverified. +[Web #63](https://github.com/MeshCore-Beacon/beacon-web/pull/63), `01c6d1aae1632b5079299a50e7ec75053138fa78`, translates RF / Signal headings, chart legends/descriptions, exact tables and explanatory/error/empty text. Shared analytics section/range controls and generic chart states also use the catalogs. English remains the default; query keys, units, numerical series, UTC windows and canonical links stay unchanged. A regression confirms that changing language makes no extra data request. -This independent PR does not expand the frozen consolidation release. The Pi runs server `c02317a4` / web `1c77f200`; immediate frontend rollback is `b8d4dbf8`. Accepted frontend `42ba5fcb`, identical in source to accepted web `0f0a6ca5`, remains an older recovery point. The [preview changelog/source](https://canadaverse.org/beacon-dev/source.html) identifies both the accepted batch and the additional language candidate. +The exact candidate passes Windows/Pi build, lint and all **812 tests**, plus CI (existing web CodeQL is skipped). Public French stays EN DIRECT; chart descriptions and exact tables translate, `7 j` keeps `range=7d`, and 320px checks show no page overflow. Phone fixes keep period labels together, separate table columns and hide crowded coverage-axis labels. The source/assets match the public offer; all 23 service containers and the accepted server were preserved. Initial JS grows by 2,153 gzip bytes with no new dependency or translation request. + +The Pi runs server `c02317a4` / web `01c6d1aa`; immediate frontend rollback is `1c77f200`. Other detailed pages, dialogs and locale-aware date/number formatting remain follow-ups under #12. Automatic chart time formatting is unchanged; physical Safari is untested. The [changelog/source](https://canadaverse.org/beacon-dev/source.html) identifies the running candidate. This is an independent translation slice, not another analytics endpoint/page or a full-parity release claim. ## Delivered foundations @@ -76,14 +78,14 @@ The September 20 review correction moves both new aggregate APIs onto materializ September 20 validation covered native Go/PostgreSQL/HTTP behavior and **786 web tests**, plus private backup compatibility, feature-only startup failure, TLS/password files, cancellation and schema/data/sequence restoration. That review update passed 390/1280px browser checks, with ten distinct glyphs and complete-hour text; earlier chart checks also covered 320/768px. This is historical evidence for the unchanged feature code. Current revisions and corresponding-source archives are on the preview's changelog page. -The September 24 consolidation check built accepted server `c02317a4` and retained web `42ba5fcb`, identical in source to accepted `0f0a6ca5`. Its native PostgreSQL and public/browser evidence remains in the release checklist. The current Pi frontend additionally includes #62 as documented above; the accepted frontend is retained for rollback. The three-hour retained analytics sample does not establish 7/30-day history or production capacity. +The September 24 consolidation check built accepted server `c02317a4` and retained web `42ba5fcb`, identical in source to accepted `0f0a6ca5`. Its native PostgreSQL and public/browser evidence remains in the release checklist. The Pi frontend now includes #63 as documented above; accepted #62 is retained for immediate rollback. The three-hour retained analytics sample does not establish 7/30-day history or production capacity. ## Next phases The September 20 #116 investigation has a new [current-build result](https://github.com/MeshCore-Beacon/beacon-server/issues/116#issuecomment-5753626821): a 600-second unmodified Pi capture kept both feeds connected and retained 2,169 new observations, with no ping timeout, disconnect, deadline, SQLSTATE error or HTTP 5xx response. App/PostgreSQL CPU averaged 2.14%/3.96% of one core. The preceding 3h39 log likewise has no MQTT loss or deadline error. Timestamp warnings were classified separately. This did not measure callback or pool-acquisition duration and does not establish the original cause or production capacity. No application, ordering, acknowledgement or service change was made; #116 remains open. Further capture should follow a recurrence or meaningful workload change, rather than repeatedly sampling the same healthy state. 1. **Publish the coordinated consolidation releases.** Accepted-dev native/Pi/public validation and the changelog/source handoff are complete. Maintainers choose versions, create signed release commits/tags, promote main and verify release artifacts. Follow [RELEASE-CHECKLIST.md](RELEASE-CHECKLIST.md). Pause new analytics until this breakpoint; this release does not claim complete CoreScope parity. -2. **Continue listed issue #12 in small screen groups.** The English/French foundation is ready in #62. Translate remaining page, chart and dialog text while preserving identifiers and measurement semantics; keep partial coverage explicit. Refresh all open issues/reviews before choosing the next group. +2. **Continue listed issue #12 in small screen groups.** The English/French foundation #62 is merged; RF / Signal is ready in #63. Translate Paths & Hashes next, preserving identifiers and measurement semantics. Other page, chart and dialog text remains; keep partial coverage explicit. Refresh all open issues/reviews before choosing the next group. 3. **After the consolidation release: Channel Activity analytics.** Inspect existing aggregates and add bounded channel traffic/trend views. Define messages versus receptions, unknown/encrypted-channel coverage and the behavior when keys are unavailable. Keep keys and message contents out of aggregate responses; avoid raw-message paging to build statistics. 4. **Observed ambiguity and topology.** Separate static prefix conflicts from observed unresolved/ambiguous paths. Add justified route-pattern, neighbour, hop and distance views with clear provenance and bounded work. 5. **Administration and backup slices.** Non-destructive archive validation is merged in #160. Resolve browser login/session and restore authorization before an import endpoint; keep deployment-file coverage and remote/scheduled backup separate. Test restoration against disposable databases. These follow-ups do not delay the current release. @@ -97,7 +99,7 @@ The September 20 #116 investigation has a new [current-build result](https://git | [Server #99](https://github.com/MeshCore-Beacon/beacon-server/issues/99) | Advert names and ACK/TRACE/PING references are accepted; define any remaining packet-type formats | | [Server #60](https://github.com/MeshCore-Beacon/beacon-server/issues/60) | Remaining administration/worker/persistence behavior; account records do not establish login sessions | | [Server #72](https://github.com/MeshCore-Beacon/beacon-server/issues/72) | Download #154 and archive validation #160 are accepted; import, browser access, deployment-file coverage and remote/scheduled backup remain | -| [Web #12](https://github.com/MeshCore-Beacon/beacon-web/issues/12) | English/French navigation foundation #62 is in review; detailed screens, chart/dialog text and formatting remain | +| [Web #12](https://github.com/MeshCore-Beacon/beacon-web/issues/12) | Foundation #62 is merged; Signal/analytics controls #63 is in review. Other screens, chart/dialog text and formatting remain | The six completed analytics/icon/map issues are closed. Refresh the five remaining issues first at every continuation; accepted partial contributions are not grounds to close broader issues. Use closing references only when a PR completes the issue's accepted scope; use related references for partial work. From 143641dd8c94ae5e35617bd3ac393c22ceaf9054 Mon Sep 17 00:00:00 2001 From: n30nex Date: Thu, 24 Sep 2026 16:32:32 -0400 Subject: [PATCH 14/69] docs(roadmap): record French paths analytics delivery --- RELEASE-CHECKLIST.md | 8 ++++---- ROADMAP.md | 20 +++++++++++--------- 2 files changed, 15 insertions(+), 13 deletions(-) diff --git a/RELEASE-CHECKLIST.md b/RELEASE-CHECKLIST.md index c46ed86..d51d5fb 100644 --- a/RELEASE-CHECKLIST.md +++ b/RELEASE-CHECKLIST.md @@ -26,7 +26,7 @@ The web source tree is identical to tested preview `42ba5fcb`; preserve that art - [x] Refresh #55/#57 after the accepted #52/#53 squash. Their source trees were identical after the September 20 refresh; that history-only update needs no replacement Pi artifact. - [x] All eight application PRs pass their required checks on the published heads. Native PostgreSQL tests ran. The upstream web CodeQL job remains skipped under its existing policy and is not counted as a scan. -All six server PRs and four web PRs have merged. Server #156/#158 and web #54/#56/#58/#60 are closed. Docs #5 remains open; language foundation #62 has since merged, and new translation #63 is outside this original frozen batch. The helper retired accepted parents/overlays without rebasing or force-pushing any application branch. Future work starts from freshly fetched dev. Backend endpoints still need deployment before dependent pages on each operator's target; Pi evidence is not evidence of another deployment. +All six server PRs and four web PRs have merged. Server #156/#158 and web #54/#56/#58/#60 are closed. Docs #5 remains open; language foundation #62 has since merged, and translations #63/#64 remain outside this original frozen batch. The helper retired accepted parents/overlays without rebasing or force-pushing any application branch. Future work starts from freshly fetched dev. Backend endpoints still need deployment before dependent pages on each operator's target; Pi evidence is not evidence of another deployment. ## Storage and retention boundary @@ -36,7 +36,7 @@ The stated replacement direction is lz4 compression, batched deletes, per-table A consolidation release must document its actual retention behavior and capacity limits. A future production parity cutover also needs verified durable history coverage and recovery copies. Do not infer either from example configuration or the Pi's short history. -Language foundation #62 has since merged as accepted web dev `6d3edbb6`, source-equivalent to tested build `1c77f200`. The live preview now additionally includes RF / Signal translation #63 at `01c6d1aa`, with 812 passing native/Windows tests and public browser/source verification. Immediate frontend rollback is `1c77f200`. The original release freeze below remains separately recorded; maintainers choose whether to advance it to include these later contributions. +Language foundation #62 has since merged as accepted web dev `6d3edbb6`, source-equivalent to tested build `1c77f200`. The live preview now includes RF / Signal #63 and Paths & Hashes #64 at `3f2a3636`, with 820 passing native/Windows tests and public browser/source verification. Immediate frontend rollback is `01c6d1aa`; merge #63 before #64. The original release freeze below remains separately recorded; maintainers choose whether to advance it to include these later contributions. ## Accepted-dev verification - 24 September @@ -45,7 +45,7 @@ Language foundation #62 has since merged as accepted web dev `6d3edbb6`, source- - [x] Accepted server is running on the preview. Source/asset hashes match; both feeds advance. Signal/Paths reconcile with SQL for global/regional 1/7/30-day selections, at 2-19 ms origin latency. Only three complete hours are populated; this does not prove 7/30-day history coverage. - [x] Browser Signal/Paths charts and map load with LIVE status and no captured warnings/errors. Unchanged frontend assets keep their actual `42ba5fcb` build/source identity and prior 786-test evidence; accepted `0f0a6ca5` has the identical tree. - [x] Only the Beacon app restarted; the other 22 containers and configuration/migration journal were preserved. Immediate rollback is server `5848d200` with unchanged web. The separate verifier retains its real `262eae96` binary/source identity. -- [x] Accepted review queues/overlays were retired. Start created the independent language branch directly from accepted dev; #63 is now the only active web entry. Completed issues are closed; five broader issues remain open. +- [x] Accepted review queues/overlays were retired. Start created the independent language branch directly from accepted dev; #63 followed by #64 is now the active web queue. Completed issues are closed; five broader issues remain open. - [ ] Maintainer version selection, signed release commits, main promotion, tags and tag-built artifact verification. Stable releases remain v1.6.0 / v1.3.0. ## Earlier combined-candidate evidence - 20 September @@ -58,7 +58,7 @@ Language foundation #62 has since merged as accepted web dev `6d3edbb6`, source- - [x] Real preview analytics reconcile with SQL for the materialized window. Browser charts, complete-hour text, small screens and error/empty/retry states are verified. Both MQTT feeds advance and the public browser reports LIVE. - [x] Current and rollback server/web artifacts, exact source offers and visible changelog match the running pair. Only the Beacon app restarted; the other 20 containers were preserved. Additive rollup migrations retain observations and are compatible with the previous binary. -The current immediate rollback restores frontend `1c77f200` with server `c02317a4`. Restore accepted frontend `42ba5fcb` before using the older consolidation server rollback to `5848d200`; its metadata describes the accepted frontend. The September 20 packet-reference rollback to `6be0f762` is an older recovery point. Exact artifacts/runners are retained; application rollback keeps additive rollup views and does not remove history. +The current immediate rollback restores frontend `01c6d1aa` with server `c02317a4`. Restore accepted frontend `42ba5fcb` before using the older consolidation server rollback to `5848d200`; its metadata describes the accepted frontend. The September 20 packet-reference rollback to `6be0f762` is an older recovery point. Exact artifacts/runners are retained; application rollback keeps additive rollup views and does not remove history. ## Maintainer release handoff diff --git a/ROADMAP.md b/ROADMAP.md index c0a7843..200f817 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -25,7 +25,7 @@ Current sites: ## Accepted consolidation batch -All ten server/web contributions merged into `dev` on 24 September UTC. The accepted batch has no remaining rebase dependencies. Language foundation [web #62](https://github.com/MeshCore-Beacon/beacon-web/pull/62) has also merged as `6d3edbb6`. Only the new Signal translation [web #63](https://github.com/MeshCore-Beacon/beacon-web/pull/63) and [docs #5](https://github.com/MeshCore-Beacon/beacon-docs/pull/5) remain open. +All ten server/web contributions merged into `dev` on 24 September UTC. The accepted batch has no remaining rebase dependencies. Language foundation [web #62](https://github.com/MeshCore-Beacon/beacon-web/pull/62) has also merged as `6d3edbb6`. Signal translation [web #63](https://github.com/MeshCore-Beacon/beacon-web/pull/63), Paths translation [web #64](https://github.com/MeshCore-Beacon/beacon-web/pull/64), and [docs #5](https://github.com/MeshCore-Beacon/beacon-docs/pull/5) remain open. The original consolidation freeze is server `c02317a4ac7228d19cab498edfa1d61186c84626` / web `0f0a6ca51c7b2c3315db77954f61b30bbdeea5e2`. CI and image builds pass at those heads; web CodeQL remains skipped and is not a security scan. Current web dev has since advanced to `6d3edbb6` through #62. Stable releases are still server **v1.6.0** and web **v1.3.0**. Merged dev is not a published stable release. @@ -44,17 +44,19 @@ The original consolidation freeze is server `c02317a4ac7228d19cab498edfa1d61186c Server #156/#158 and web #54/#56/#58/#60 are closed. Server #60/#72/#99/#116 and web #12 remain open: the accepted work completes slices, not their remaining scope. Archive verification establishes structure and integrity, not authenticity, SQL safety or restorability. Packet-carried ACK/TRACE/PING references are not identity or delivery guarantees. -The workflow removed the accepted entries and overlays without rebasing or force-pushing any application branch. Signal translation #63 started directly from fresh dev `6d3edbb6` and is the sole active web entry; future overlapping follow-ups use that declared parent through the helper. Preserve actual build identities when accepted commits have identical source trees. See [the contributor workflow](CONTRIBUTOR_WORKFLOW.md) and [consolidation checklist](RELEASE-CHECKLIST.md). +The workflow removed the accepted entries and overlays without rebasing or force-pushing any application branch. Signal translation #63 started directly from fresh dev `6d3edbb6`; Paths translation #64 follows its exact `01c6d1aa` head. Merge #63 before #64. Future overlapping follow-ups use the declared parent through the helper. Preserve actual build identities when accepted commits have identical source trees. See [the contributor workflow](CONTRIBUTOR_WORKFLOW.md) and [consolidation checklist](RELEASE-CHECKLIST.md). ## Current independent issue work -Language foundation #62 is merged as accepted dev `6d3edbb61bb79fe7aa336c5ae93a6c0a3f6f7545`. Its accepted tree equals the tested `1c77f200` build, so the merge needed no rebase, artifact relabel or Pi rebuild. +Language foundation #62 is merged as accepted dev `6d3edbb61bb79fe7aa336c5ae93a6c0a3f6f7545`, source-equivalent to tested `1c77f200`. Signal translation [#63](https://github.com/MeshCore-Beacon/beacon-web/pull/63), `01c6d1aa`, remains in review with 812 passing Windows/Pi tests. It translates RF / Signal and shared analytics controls. -[Web #63](https://github.com/MeshCore-Beacon/beacon-web/pull/63), `01c6d1aae1632b5079299a50e7ec75053138fa78`, translates RF / Signal headings, chart legends/descriptions, exact tables and explanatory/error/empty text. Shared analytics section/range controls and generic chart states also use the catalogs. English remains the default; query keys, units, numerical series, UTC windows and canonical links stay unchanged. A regression confirms that changing language makes no extra data request. +[Web #64](https://github.com/MeshCore-Beacon/beacon-web/pull/64), `3f2a36361cefb97751239aa1b82730a2b2585f0a`, translates Paths & Hashes: headings, chart names/legends/descriptions, width labels, classifications, exact tables and explanatory/loading/error/empty text. **Merge #63 before #64**; its exact parent is `01c6d1aae1632b5079299a50e7ec75053138fa78`. The helper reuses both verified trees without rebasing or rewriting branches. -The exact candidate passes Windows/Pi build, lint and all **812 tests**, plus CI (existing web CodeQL is skipped). Public French stays EN DIRECT; chart descriptions and exact tables translate, `7 j` keeps `range=7d`, and 320px checks show no page overflow. Phone fixes keep period labels together, separate table columns and hide crowded coverage-axis labels. The source/assets match the public offer; all 23 service containers and the accepted server were preserved. Initial JS grows by 2,153 gzip bytes with no new dependency or translation request. +The exact candidate passes Windows/Pi build, lint and **820 tests in 94 files**, plus CI (existing CodeQL is skipped). Language switching updates labels and keeps open details while preserving numerical series, hash widths, UTC windows, recorded zeroes versus missing hours, query keys and canonical links. A real query-cache regression proves no extra data request on a language-only change. -The Pi runs server `c02317a4` / web `01c6d1aa`; immediate frontend rollback is `1c77f200`. Other detailed pages, dialogs and locale-aware date/number formatting remain follow-ups under #12. Automatic chart time formatting is unchanged; physical Safari is untested. The [changelog/source](https://canadaverse.org/beacon-dev/source.html) identifies the running candidate. This is an independent translation slice, not another analytics endpoint/page or a full-parity release claim. +Public French stays EN DIRECT, preserves classification counts during switching, and keeps `range=7d` for `7 j`. Charts, descriptions and exact tables fit a 320px viewport with contained hourly-table scrolling; numeric columns have explicit spacing. Source/JS/CSS hashes match. All 23 containers and the accepted server were preserved. Initial JS increases by 1,456 gzip bytes over #63 with no dependency or translation request added. + +The Pi runs server `c02317a4` / web `3f2a3636`; immediate frontend rollback is `01c6d1aa`. Current source and rollback archives remain available, and completed staging was retired. The [preview changelog/source](https://canadaverse.org/beacon-dev/source.html) identifies the running candidate. Other detailed pages, dialogs and locale-aware date/number formatting remain follow-ups under #12; automatic chart time formatting is unchanged. Physical Safari remains untested. These translation slices add no analytics endpoint/page and do not change the owner-managed release freeze. ## Delivered foundations @@ -78,14 +80,14 @@ The September 20 review correction moves both new aggregate APIs onto materializ September 20 validation covered native Go/PostgreSQL/HTTP behavior and **786 web tests**, plus private backup compatibility, feature-only startup failure, TLS/password files, cancellation and schema/data/sequence restoration. That review update passed 390/1280px browser checks, with ten distinct glyphs and complete-hour text; earlier chart checks also covered 320/768px. This is historical evidence for the unchanged feature code. Current revisions and corresponding-source archives are on the preview's changelog page. -The September 24 consolidation check built accepted server `c02317a4` and retained web `42ba5fcb`, identical in source to accepted `0f0a6ca5`. Its native PostgreSQL and public/browser evidence remains in the release checklist. The Pi frontend now includes #63 as documented above; accepted #62 is retained for immediate rollback. The three-hour retained analytics sample does not establish 7/30-day history or production capacity. +The September 24 consolidation check built accepted server `c02317a4` and retained web `42ba5fcb`, identical in source to accepted `0f0a6ca5`. Its native PostgreSQL and public/browser evidence remains in the release checklist. The Pi frontend now includes #63/#64 as documented above; #63 is retained for immediate rollback. The three-hour retained analytics sample does not establish 7/30-day history or production capacity. ## Next phases The September 20 #116 investigation has a new [current-build result](https://github.com/MeshCore-Beacon/beacon-server/issues/116#issuecomment-5753626821): a 600-second unmodified Pi capture kept both feeds connected and retained 2,169 new observations, with no ping timeout, disconnect, deadline, SQLSTATE error or HTTP 5xx response. App/PostgreSQL CPU averaged 2.14%/3.96% of one core. The preceding 3h39 log likewise has no MQTT loss or deadline error. Timestamp warnings were classified separately. This did not measure callback or pool-acquisition duration and does not establish the original cause or production capacity. No application, ordering, acknowledgement or service change was made; #116 remains open. Further capture should follow a recurrence or meaningful workload change, rather than repeatedly sampling the same healthy state. 1. **Publish the coordinated consolidation releases.** Accepted-dev native/Pi/public validation and the changelog/source handoff are complete. Maintainers choose versions, create signed release commits/tags, promote main and verify release artifacts. Follow [RELEASE-CHECKLIST.md](RELEASE-CHECKLIST.md). Pause new analytics until this breakpoint; this release does not claim complete CoreScope parity. -2. **Continue listed issue #12 in small screen groups.** The English/French foundation #62 is merged; RF / Signal is ready in #63. Translate Paths & Hashes next, preserving identifiers and measurement semantics. Other page, chart and dialog text remains; keep partial coverage explicit. Refresh all open issues/reviews before choosing the next group. +2. **Continue listed issue #12 in small screen groups.** The English/French foundation #62 is merged; RF / Signal (#63) and Paths & Hashes (#64) are ready for review. Translate Traffic next, preserving identifiers and measurement semantics. Other page, chart and dialog text remains; keep partial coverage explicit. Refresh all open issues/reviews before choosing the next group. 3. **After the consolidation release: Channel Activity analytics.** Inspect existing aggregates and add bounded channel traffic/trend views. Define messages versus receptions, unknown/encrypted-channel coverage and the behavior when keys are unavailable. Keep keys and message contents out of aggregate responses; avoid raw-message paging to build statistics. 4. **Observed ambiguity and topology.** Separate static prefix conflicts from observed unresolved/ambiguous paths. Add justified route-pattern, neighbour, hop and distance views with clear provenance and bounded work. 5. **Administration and backup slices.** Non-destructive archive validation is merged in #160. Resolve browser login/session and restore authorization before an import endpoint; keep deployment-file coverage and remote/scheduled backup separate. Test restoration against disposable databases. These follow-ups do not delay the current release. @@ -99,7 +101,7 @@ The September 20 #116 investigation has a new [current-build result](https://git | [Server #99](https://github.com/MeshCore-Beacon/beacon-server/issues/99) | Advert names and ACK/TRACE/PING references are accepted; define any remaining packet-type formats | | [Server #60](https://github.com/MeshCore-Beacon/beacon-server/issues/60) | Remaining administration/worker/persistence behavior; account records do not establish login sessions | | [Server #72](https://github.com/MeshCore-Beacon/beacon-server/issues/72) | Download #154 and archive validation #160 are accepted; import, browser access, deployment-file coverage and remote/scheduled backup remain | -| [Web #12](https://github.com/MeshCore-Beacon/beacon-web/issues/12) | Foundation #62 is merged; Signal/analytics controls #63 is in review. Other screens, chart/dialog text and formatting remain | +| [Web #12](https://github.com/MeshCore-Beacon/beacon-web/issues/12) | Foundation #62 is merged; Signal/analytics controls #63 and Paths #64 are in review. Traffic and other screens, chart/dialog text and formatting remain | The six completed analytics/icon/map issues are closed. Refresh the five remaining issues first at every continuation; accepted partial contributions are not grounds to close broader issues. Use closing references only when a PR completes the issue's accepted scope; use related references for partial work. From ee078fbb0ea87053366a6c613bc7d479b824f7f6 Mon Sep 17 00:00:00 2001 From: n30nex Date: Thu, 24 Sep 2026 23:01:13 -0400 Subject: [PATCH 15/69] docs(roadmap): record French traffic analytics delivery --- RELEASE-CHECKLIST.md | 8 ++++---- ROADMAP.md | 22 +++++++++++----------- 2 files changed, 15 insertions(+), 15 deletions(-) diff --git a/RELEASE-CHECKLIST.md b/RELEASE-CHECKLIST.md index d51d5fb..24b86f7 100644 --- a/RELEASE-CHECKLIST.md +++ b/RELEASE-CHECKLIST.md @@ -26,7 +26,7 @@ The web source tree is identical to tested preview `42ba5fcb`; preserve that art - [x] Refresh #55/#57 after the accepted #52/#53 squash. Their source trees were identical after the September 20 refresh; that history-only update needs no replacement Pi artifact. - [x] All eight application PRs pass their required checks on the published heads. Native PostgreSQL tests ran. The upstream web CodeQL job remains skipped under its existing policy and is not counted as a scan. -All six server PRs and four web PRs have merged. Server #156/#158 and web #54/#56/#58/#60 are closed. Docs #5 remains open; language foundation #62 has since merged, and translations #63/#64 remain outside this original frozen batch. The helper retired accepted parents/overlays without rebasing or force-pushing any application branch. Future work starts from freshly fetched dev. Backend endpoints still need deployment before dependent pages on each operator's target; Pi evidence is not evidence of another deployment. +All six server PRs and four web PRs have merged. Server #156/#158 and web #54/#56/#58/#60 are closed. Docs #5 remains open; language foundation #62 has since merged, and translations #63/#64/#65 remain outside this original frozen batch. The helper retired accepted parents/overlays without rebasing or force-pushing any application branch. Future work starts from freshly fetched dev. Backend endpoints still need deployment before dependent pages on each operator's target; Pi evidence is not evidence of another deployment. ## Storage and retention boundary @@ -36,7 +36,7 @@ The stated replacement direction is lz4 compression, batched deletes, per-table A consolidation release must document its actual retention behavior and capacity limits. A future production parity cutover also needs verified durable history coverage and recovery copies. Do not infer either from example configuration or the Pi's short history. -Language foundation #62 has since merged as accepted web dev `6d3edbb6`, source-equivalent to tested build `1c77f200`. The live preview now includes RF / Signal #63 and Paths & Hashes #64 at `3f2a3636`, with 820 passing native/Windows tests and public browser/source verification. Immediate frontend rollback is `01c6d1aa`; merge #63 before #64. The original release freeze below remains separately recorded; maintainers choose whether to advance it to include these later contributions. +Language foundation #62 has since merged as accepted web dev `6d3edbb6`, source-equivalent to tested build `1c77f200`. The live preview now includes Signal #63, Paths #64 and Traffic #65 at `a928fda5`, with 828 passing native/Windows tests and public browser/source verification. Immediate frontend rollback is `3f2a3636`; merge #63, then #64, then #65. The original release freeze below remains separately recorded; maintainers choose whether to advance it to include these later contributions. ## Accepted-dev verification - 24 September @@ -45,7 +45,7 @@ Language foundation #62 has since merged as accepted web dev `6d3edbb6`, source- - [x] Accepted server is running on the preview. Source/asset hashes match; both feeds advance. Signal/Paths reconcile with SQL for global/regional 1/7/30-day selections, at 2-19 ms origin latency. Only three complete hours are populated; this does not prove 7/30-day history coverage. - [x] Browser Signal/Paths charts and map load with LIVE status and no captured warnings/errors. Unchanged frontend assets keep their actual `42ba5fcb` build/source identity and prior 786-test evidence; accepted `0f0a6ca5` has the identical tree. - [x] Only the Beacon app restarted; the other 22 containers and configuration/migration journal were preserved. Immediate rollback is server `5848d200` with unchanged web. The separate verifier retains its real `262eae96` binary/source identity. -- [x] Accepted review queues/overlays were retired. Start created the independent language branch directly from accepted dev; #63 followed by #64 is now the active web queue. Completed issues are closed; five broader issues remain open. +- [x] Accepted review queues/overlays were retired. Start created the independent language branch directly from accepted dev; #63 followed by #64 and #65 is now the active web queue. Completed issues are closed; five broader issues remain open. - [ ] Maintainer version selection, signed release commits, main promotion, tags and tag-built artifact verification. Stable releases remain v1.6.0 / v1.3.0. ## Earlier combined-candidate evidence - 20 September @@ -58,7 +58,7 @@ Language foundation #62 has since merged as accepted web dev `6d3edbb6`, source- - [x] Real preview analytics reconcile with SQL for the materialized window. Browser charts, complete-hour text, small screens and error/empty/retry states are verified. Both MQTT feeds advance and the public browser reports LIVE. - [x] Current and rollback server/web artifacts, exact source offers and visible changelog match the running pair. Only the Beacon app restarted; the other 20 containers were preserved. Additive rollup migrations retain observations and are compatible with the previous binary. -The current immediate rollback restores frontend `01c6d1aa` with server `c02317a4`. Restore accepted frontend `42ba5fcb` before using the older consolidation server rollback to `5848d200`; its metadata describes the accepted frontend. The September 20 packet-reference rollback to `6be0f762` is an older recovery point. Exact artifacts/runners are retained; application rollback keeps additive rollup views and does not remove history. +The current immediate rollback restores frontend `3f2a3636` with server `c02317a4`. Restore accepted frontend `42ba5fcb` before using the older consolidation server rollback to `5848d200`; its metadata describes the accepted frontend. The September 20 packet-reference rollback to `6be0f762` is an older recovery point. Exact artifacts/runners are retained; application rollback keeps additive rollup views and does not remove history. ## Maintainer release handoff diff --git a/ROADMAP.md b/ROADMAP.md index 200f817..532df65 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,6 +1,6 @@ # Beacon parity and analytics roadmap -Updated 24 September 2026. This is the working roadmap for n30nex's ongoing contributions toward CoreScope feature parity. Maintainers decide acceptance and merge order; deployment owners handle the production switch. +Updated 25 September 2026 UTC. This is the working roadmap for n30nex's ongoing contributions toward CoreScope feature parity. Maintainers decide acceptance and merge order; deployment owners handle the production switch. Refresh GitHub issues, PR feedback and branch state before starting a phase. This document is a snapshot, and linked issues/PRs are the current source of truth. @@ -25,7 +25,7 @@ Current sites: ## Accepted consolidation batch -All ten server/web contributions merged into `dev` on 24 September UTC. The accepted batch has no remaining rebase dependencies. Language foundation [web #62](https://github.com/MeshCore-Beacon/beacon-web/pull/62) has also merged as `6d3edbb6`. Signal translation [web #63](https://github.com/MeshCore-Beacon/beacon-web/pull/63), Paths translation [web #64](https://github.com/MeshCore-Beacon/beacon-web/pull/64), and [docs #5](https://github.com/MeshCore-Beacon/beacon-docs/pull/5) remain open. +All ten server/web contributions merged into `dev` on 24 September UTC. The accepted batch has no remaining rebase dependencies. Language foundation [web #62](https://github.com/MeshCore-Beacon/beacon-web/pull/62) has also merged as `6d3edbb6`. Signal translation [web #63](https://github.com/MeshCore-Beacon/beacon-web/pull/63), Paths translation [web #64](https://github.com/MeshCore-Beacon/beacon-web/pull/64), Traffic translation [web #65](https://github.com/MeshCore-Beacon/beacon-web/pull/65), and [docs #5](https://github.com/MeshCore-Beacon/beacon-docs/pull/5) remain open. The original consolidation freeze is server `c02317a4ac7228d19cab498edfa1d61186c84626` / web `0f0a6ca51c7b2c3315db77954f61b30bbdeea5e2`. CI and image builds pass at those heads; web CodeQL remains skipped and is not a security scan. Current web dev has since advanced to `6d3edbb6` through #62. Stable releases are still server **v1.6.0** and web **v1.3.0**. Merged dev is not a published stable release. @@ -44,19 +44,19 @@ The original consolidation freeze is server `c02317a4ac7228d19cab498edfa1d61186c Server #156/#158 and web #54/#56/#58/#60 are closed. Server #60/#72/#99/#116 and web #12 remain open: the accepted work completes slices, not their remaining scope. Archive verification establishes structure and integrity, not authenticity, SQL safety or restorability. Packet-carried ACK/TRACE/PING references are not identity or delivery guarantees. -The workflow removed the accepted entries and overlays without rebasing or force-pushing any application branch. Signal translation #63 started directly from fresh dev `6d3edbb6`; Paths translation #64 follows its exact `01c6d1aa` head. Merge #63 before #64. Future overlapping follow-ups use the declared parent through the helper. Preserve actual build identities when accepted commits have identical source trees. See [the contributor workflow](CONTRIBUTOR_WORKFLOW.md) and [consolidation checklist](RELEASE-CHECKLIST.md). +The workflow removed the accepted entries and overlays without rebasing or force-pushing any application branch. Signal translation #63 started directly from fresh dev `6d3edbb6`; Paths translation #64 follows its exact `01c6d1aa` head. Traffic translation #65 follows #64 at `3f2a3636`. Merge #63, then #64, then #65. Future overlapping follow-ups use the declared parent through the helper. Preserve actual build identities when accepted commits have identical source trees. See [the contributor workflow](CONTRIBUTOR_WORKFLOW.md) and [consolidation checklist](RELEASE-CHECKLIST.md). ## Current independent issue work -Language foundation #62 is merged as accepted dev `6d3edbb61bb79fe7aa336c5ae93a6c0a3f6f7545`, source-equivalent to tested `1c77f200`. Signal translation [#63](https://github.com/MeshCore-Beacon/beacon-web/pull/63), `01c6d1aa`, remains in review with 812 passing Windows/Pi tests. It translates RF / Signal and shared analytics controls. +Language foundation #62 is merged as accepted dev `6d3edbb61bb79fe7aa336c5ae93a6c0a3f6f7545`, source-equivalent to tested `1c77f200`. Signal/shared-controls #63 (`01c6d1aa`, 812 tests) and Paths & Hashes #64 (`3f2a3636`, 820 tests) remain in review. -[Web #64](https://github.com/MeshCore-Beacon/beacon-web/pull/64), `3f2a36361cefb97751239aa1b82730a2b2585f0a`, translates Paths & Hashes: headings, chart names/legends/descriptions, width labels, classifications, exact tables and explanatory/loading/error/empty text. **Merge #63 before #64**; its exact parent is `01c6d1aae1632b5079299a50e7ec75053138fa78`. The helper reuses both verified trees without rebasing or rewriting branches. +[Web #65](https://github.com/MeshCore-Beacon/beacon-web/pull/65), `a928fda5b34d2663ea9a970a43bafd0406a4a133`, translates Traffic: trend/share/heatmap labels and descriptions, grouped/unassigned display labels, tooltip text, the exact IATA table and explanatory/loading/error/empty states. Exact parent is #64 at `3f2a36361cefb97751239aa1b82730a2b2585f0a`. **Merge #63, then #64, then #65.** The helper reuses validated trees without rewriting branches. -The exact candidate passes Windows/Pi build, lint and **820 tests in 94 files**, plus CI (existing CodeQL is skipped). Language switching updates labels and keeps open details while preserving numerical series, hash widths, UTC windows, recorded zeroes versus missing hours, query keys and canonical links. A real query-cache regression proves no extra data request on a language-only change. +All **828 tests in 94 files**, build and lint pass on Windows and the native Pi; exact-head CI passes (existing CodeQL is skipped). Actual IATA codes, model identifiers, reception counts, grouping/sorting, recorded zeroes versus gaps, UTC/partial-hour behavior, query keys and canonical links stay unchanged. Tooltip plural selection uses the raw count while showing the existing formatted number. A language-only switch makes no additional data request. -Public French stays EN DIRECT, preserves classification counts during switching, and keeps `range=7d` for `7 j`. Charts, descriptions and exact tables fit a 320px viewport with contained hourly-table scrolling; numeric columns have explicit spacing. Source/JS/CSS hashes match. All 23 containers and the accepted server were preserved. Initial JS increases by 1,456 gzip bytes over #63 with no dependency or translation request added. +Public French stays EN DIRECT; all 27 displayed IATA rows and values match across language switching. Desktop/320px charts and the exact table fit, with spaced numeric columns and no page overflow. Local browser checks cover the French heatmap tooltip, and public `7 j` retains `range=7d`. Source/JS/CSS hashes match, both feeds are connected and all 23 containers are preserved. Initial JS grows by 863 gzip bytes over #64 with no dependency or translation request added. -The Pi runs server `c02317a4` / web `3f2a3636`; immediate frontend rollback is `01c6d1aa`. Current source and rollback archives remain available, and completed staging was retired. The [preview changelog/source](https://canadaverse.org/beacon-dev/source.html) identifies the running candidate. Other detailed pages, dialogs and locale-aware date/number formatting remain follow-ups under #12; automatic chart time formatting is unchanged. Physical Safari remains untested. These translation slices add no analytics endpoint/page and do not change the owner-managed release freeze. +Current Pi: server `c02317a4` / web `a928fda5`. Immediate frontend rollback is `3f2a3636`; source and distributions for both remain available, and completed staging was retired. The [preview changelog/source](https://canadaverse.org/beacon-dev/source.html) identifies the build. Other pages/dialogs and locale-aware data formatting remain under #12; physical Safari is untested. These translation slices add no analytics endpoint/page and leave the original owner-managed release freeze separate. ## Delivered foundations @@ -80,14 +80,14 @@ The September 20 review correction moves both new aggregate APIs onto materializ September 20 validation covered native Go/PostgreSQL/HTTP behavior and **786 web tests**, plus private backup compatibility, feature-only startup failure, TLS/password files, cancellation and schema/data/sequence restoration. That review update passed 390/1280px browser checks, with ten distinct glyphs and complete-hour text; earlier chart checks also covered 320/768px. This is historical evidence for the unchanged feature code. Current revisions and corresponding-source archives are on the preview's changelog page. -The September 24 consolidation check built accepted server `c02317a4` and retained web `42ba5fcb`, identical in source to accepted `0f0a6ca5`. Its native PostgreSQL and public/browser evidence remains in the release checklist. The Pi frontend now includes #63/#64 as documented above; #63 is retained for immediate rollback. The three-hour retained analytics sample does not establish 7/30-day history or production capacity. +The September 24 consolidation check built accepted server `c02317a4` and retained web `42ba5fcb`, identical in source to accepted `0f0a6ca5`. Its native PostgreSQL and public/browser evidence remains in the release checklist. The Pi frontend now includes #63/#64/#65 as documented above; #64 is retained for immediate rollback. The three-hour retained analytics sample does not establish 7/30-day history or production capacity. ## Next phases The September 20 #116 investigation has a new [current-build result](https://github.com/MeshCore-Beacon/beacon-server/issues/116#issuecomment-5753626821): a 600-second unmodified Pi capture kept both feeds connected and retained 2,169 new observations, with no ping timeout, disconnect, deadline, SQLSTATE error or HTTP 5xx response. App/PostgreSQL CPU averaged 2.14%/3.96% of one core. The preceding 3h39 log likewise has no MQTT loss or deadline error. Timestamp warnings were classified separately. This did not measure callback or pool-acquisition duration and does not establish the original cause or production capacity. No application, ordering, acknowledgement or service change was made; #116 remains open. Further capture should follow a recurrence or meaningful workload change, rather than repeatedly sampling the same healthy state. 1. **Publish the coordinated consolidation releases.** Accepted-dev native/Pi/public validation and the changelog/source handoff are complete. Maintainers choose versions, create signed release commits/tags, promote main and verify release artifacts. Follow [RELEASE-CHECKLIST.md](RELEASE-CHECKLIST.md). Pause new analytics until this breakpoint; this release does not claim complete CoreScope parity. -2. **Continue listed issue #12 in small screen groups.** The English/French foundation #62 is merged; RF / Signal (#63) and Paths & Hashes (#64) are ready for review. Translate Traffic next, preserving identifiers and measurement semantics. Other page, chart and dialog text remains; keep partial coverage explicit. Refresh all open issues/reviews before choosing the next group. +2. **Continue listed issue #12 in small screen groups.** The English/French foundation #62 is merged; RF / Signal (#63), Paths & Hashes (#64) and Traffic (#65) are ready for review. Translate Scopes next, preserving identifiers and measurement semantics. Other page, chart and dialog text remains; keep partial coverage explicit. Refresh all open issues/reviews before choosing the next group. 3. **After the consolidation release: Channel Activity analytics.** Inspect existing aggregates and add bounded channel traffic/trend views. Define messages versus receptions, unknown/encrypted-channel coverage and the behavior when keys are unavailable. Keep keys and message contents out of aggregate responses; avoid raw-message paging to build statistics. 4. **Observed ambiguity and topology.** Separate static prefix conflicts from observed unresolved/ambiguous paths. Add justified route-pattern, neighbour, hop and distance views with clear provenance and bounded work. 5. **Administration and backup slices.** Non-destructive archive validation is merged in #160. Resolve browser login/session and restore authorization before an import endpoint; keep deployment-file coverage and remote/scheduled backup separate. Test restoration against disposable databases. These follow-ups do not delay the current release. @@ -101,7 +101,7 @@ The September 20 #116 investigation has a new [current-build result](https://git | [Server #99](https://github.com/MeshCore-Beacon/beacon-server/issues/99) | Advert names and ACK/TRACE/PING references are accepted; define any remaining packet-type formats | | [Server #60](https://github.com/MeshCore-Beacon/beacon-server/issues/60) | Remaining administration/worker/persistence behavior; account records do not establish login sessions | | [Server #72](https://github.com/MeshCore-Beacon/beacon-server/issues/72) | Download #154 and archive validation #160 are accepted; import, browser access, deployment-file coverage and remote/scheduled backup remain | -| [Web #12](https://github.com/MeshCore-Beacon/beacon-web/issues/12) | Foundation #62 is merged; Signal/analytics controls #63 and Paths #64 are in review. Traffic and other screens, chart/dialog text and formatting remain | +| [Web #12](https://github.com/MeshCore-Beacon/beacon-web/issues/12) | Foundation #62 is merged; Signal/analytics controls #63, Paths #64 and Traffic #65 are in review. Scopes and other screens, chart/dialog text and formatting remain | The six completed analytics/icon/map issues are closed. Refresh the five remaining issues first at every continuation; accepted partial contributions are not grounds to close broader issues. Use closing references only when a PR completes the issue's accepted scope; use related references for partial work. From 19510f70c3f7b9918df5ed72709c09585a6bb09e Mon Sep 17 00:00:00 2001 From: n30nex Date: Fri, 25 Sep 2026 01:39:37 -0400 Subject: [PATCH 16/69] docs(roadmap): record French scope analytics delivery --- RELEASE-CHECKLIST.md | 8 ++++---- ROADMAP.md | 20 ++++++++++---------- 2 files changed, 14 insertions(+), 14 deletions(-) diff --git a/RELEASE-CHECKLIST.md b/RELEASE-CHECKLIST.md index 24b86f7..6b84d71 100644 --- a/RELEASE-CHECKLIST.md +++ b/RELEASE-CHECKLIST.md @@ -26,7 +26,7 @@ The web source tree is identical to tested preview `42ba5fcb`; preserve that art - [x] Refresh #55/#57 after the accepted #52/#53 squash. Their source trees were identical after the September 20 refresh; that history-only update needs no replacement Pi artifact. - [x] All eight application PRs pass their required checks on the published heads. Native PostgreSQL tests ran. The upstream web CodeQL job remains skipped under its existing policy and is not counted as a scan. -All six server PRs and four web PRs have merged. Server #156/#158 and web #54/#56/#58/#60 are closed. Docs #5 remains open; language foundation #62 has since merged, and translations #63/#64/#65 remain outside this original frozen batch. The helper retired accepted parents/overlays without rebasing or force-pushing any application branch. Future work starts from freshly fetched dev. Backend endpoints still need deployment before dependent pages on each operator's target; Pi evidence is not evidence of another deployment. +All six server PRs and four web PRs have merged. Server #156/#158 and web #54/#56/#58/#60 are closed. Docs #5 remains open; language foundation #62 has since merged, and translations #63/#64/#65/#66 remain outside this original frozen batch. The helper retired accepted parents/overlays without rebasing or force-pushing any application branch. Future work starts from freshly fetched dev. Backend endpoints still need deployment before dependent pages on each operator's target; Pi evidence is not evidence of another deployment. ## Storage and retention boundary @@ -36,7 +36,7 @@ The stated replacement direction is lz4 compression, batched deletes, per-table A consolidation release must document its actual retention behavior and capacity limits. A future production parity cutover also needs verified durable history coverage and recovery copies. Do not infer either from example configuration or the Pi's short history. -Language foundation #62 has since merged as accepted web dev `6d3edbb6`, source-equivalent to tested build `1c77f200`. The live preview now includes Signal #63, Paths #64 and Traffic #65 at `a928fda5`, with 828 passing native/Windows tests and public browser/source verification. Immediate frontend rollback is `3f2a3636`; merge #63, then #64, then #65. The original release freeze below remains separately recorded; maintainers choose whether to advance it to include these later contributions. +Language foundation #62 has since merged as accepted web dev `6d3edbb6`, source-equivalent to tested build `1c77f200`. The live preview now includes Signal #63, Paths #64, Traffic #65 and Scopes #66 at `8d81bd5d`, with 836 passing native/Windows tests and public browser/source verification. The real scope dataset is empty; populated Scopes checks use a local fixture. Immediate frontend rollback is `a928fda5`; merge #63, #64, #65, then #66. The original release freeze below remains separately recorded; maintainers choose whether to advance it to include these later contributions. ## Accepted-dev verification - 24 September @@ -45,7 +45,7 @@ Language foundation #62 has since merged as accepted web dev `6d3edbb6`, source- - [x] Accepted server is running on the preview. Source/asset hashes match; both feeds advance. Signal/Paths reconcile with SQL for global/regional 1/7/30-day selections, at 2-19 ms origin latency. Only three complete hours are populated; this does not prove 7/30-day history coverage. - [x] Browser Signal/Paths charts and map load with LIVE status and no captured warnings/errors. Unchanged frontend assets keep their actual `42ba5fcb` build/source identity and prior 786-test evidence; accepted `0f0a6ca5` has the identical tree. - [x] Only the Beacon app restarted; the other 22 containers and configuration/migration journal were preserved. Immediate rollback is server `5848d200` with unchanged web. The separate verifier retains its real `262eae96` binary/source identity. -- [x] Accepted review queues/overlays were retired. Start created the independent language branch directly from accepted dev; #63 followed by #64 and #65 is now the active web queue. Completed issues are closed; five broader issues remain open. +- [x] Accepted review queues/overlays were retired. Start created the independent language branch directly from accepted dev; #63 followed by #64, #65 and #66 is now the active web queue. Completed issues are closed; five broader issues remain open. - [ ] Maintainer version selection, signed release commits, main promotion, tags and tag-built artifact verification. Stable releases remain v1.6.0 / v1.3.0. ## Earlier combined-candidate evidence - 20 September @@ -58,7 +58,7 @@ Language foundation #62 has since merged as accepted web dev `6d3edbb6`, source- - [x] Real preview analytics reconcile with SQL for the materialized window. Browser charts, complete-hour text, small screens and error/empty/retry states are verified. Both MQTT feeds advance and the public browser reports LIVE. - [x] Current and rollback server/web artifacts, exact source offers and visible changelog match the running pair. Only the Beacon app restarted; the other 20 containers were preserved. Additive rollup migrations retain observations and are compatible with the previous binary. -The current immediate rollback restores frontend `3f2a3636` with server `c02317a4`. Restore accepted frontend `42ba5fcb` before using the older consolidation server rollback to `5848d200`; its metadata describes the accepted frontend. The September 20 packet-reference rollback to `6be0f762` is an older recovery point. Exact artifacts/runners are retained; application rollback keeps additive rollup views and does not remove history. +The current immediate rollback restores frontend `a928fda5` with server `c02317a4`. Restore accepted frontend `42ba5fcb` before using the older consolidation server rollback to `5848d200`; its metadata describes the accepted frontend. The September 20 packet-reference rollback to `6be0f762` is an older recovery point. Exact artifacts/runners are retained; application rollback keeps additive rollup views and does not remove history. ## Maintainer release handoff diff --git a/ROADMAP.md b/ROADMAP.md index 532df65..3f008cb 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -25,7 +25,7 @@ Current sites: ## Accepted consolidation batch -All ten server/web contributions merged into `dev` on 24 September UTC. The accepted batch has no remaining rebase dependencies. Language foundation [web #62](https://github.com/MeshCore-Beacon/beacon-web/pull/62) has also merged as `6d3edbb6`. Signal translation [web #63](https://github.com/MeshCore-Beacon/beacon-web/pull/63), Paths translation [web #64](https://github.com/MeshCore-Beacon/beacon-web/pull/64), Traffic translation [web #65](https://github.com/MeshCore-Beacon/beacon-web/pull/65), and [docs #5](https://github.com/MeshCore-Beacon/beacon-docs/pull/5) remain open. +All ten server/web contributions merged into `dev` on 24 September UTC. The accepted batch has no remaining rebase dependencies. Language foundation [web #62](https://github.com/MeshCore-Beacon/beacon-web/pull/62) has also merged as `6d3edbb6`. Signal translation [web #63](https://github.com/MeshCore-Beacon/beacon-web/pull/63), Paths translation [web #64](https://github.com/MeshCore-Beacon/beacon-web/pull/64), Traffic translation [web #65](https://github.com/MeshCore-Beacon/beacon-web/pull/65), Scopes translation [web #66](https://github.com/MeshCore-Beacon/beacon-web/pull/66), and [docs #5](https://github.com/MeshCore-Beacon/beacon-docs/pull/5) remain open. The original consolidation freeze is server `c02317a4ac7228d19cab498edfa1d61186c84626` / web `0f0a6ca51c7b2c3315db77954f61b30bbdeea5e2`. CI and image builds pass at those heads; web CodeQL remains skipped and is not a security scan. Current web dev has since advanced to `6d3edbb6` through #62. Stable releases are still server **v1.6.0** and web **v1.3.0**. Merged dev is not a published stable release. @@ -44,19 +44,19 @@ The original consolidation freeze is server `c02317a4ac7228d19cab498edfa1d61186c Server #156/#158 and web #54/#56/#58/#60 are closed. Server #60/#72/#99/#116 and web #12 remain open: the accepted work completes slices, not their remaining scope. Archive verification establishes structure and integrity, not authenticity, SQL safety or restorability. Packet-carried ACK/TRACE/PING references are not identity or delivery guarantees. -The workflow removed the accepted entries and overlays without rebasing or force-pushing any application branch. Signal translation #63 started directly from fresh dev `6d3edbb6`; Paths translation #64 follows its exact `01c6d1aa` head. Traffic translation #65 follows #64 at `3f2a3636`. Merge #63, then #64, then #65. Future overlapping follow-ups use the declared parent through the helper. Preserve actual build identities when accepted commits have identical source trees. See [the contributor workflow](CONTRIBUTOR_WORKFLOW.md) and [consolidation checklist](RELEASE-CHECKLIST.md). +The workflow removed the accepted entries and overlays without rebasing or force-pushing any application branch. Signal translation #63 started directly from fresh dev `6d3edbb6`; Paths translation #64 follows its exact `01c6d1aa` head. Traffic translation #65 follows #64 at `3f2a3636`. Scopes #66 follows #65 at `a928fda5`. Merge #63, #64, #65, then #66. Future overlapping follow-ups use the declared parent through the helper. Preserve actual build identities when accepted commits have identical source trees. See [the contributor workflow](CONTRIBUTOR_WORKFLOW.md) and [consolidation checklist](RELEASE-CHECKLIST.md). ## Current independent issue work -Language foundation #62 is merged as accepted dev `6d3edbb61bb79fe7aa336c5ae93a6c0a3f6f7545`, source-equivalent to tested `1c77f200`. Signal/shared-controls #63 (`01c6d1aa`, 812 tests) and Paths & Hashes #64 (`3f2a3636`, 820 tests) remain in review. +Language foundation #62 is merged as accepted dev `6d3edbb61bb79fe7aa336c5ae93a6c0a3f6f7545`, source-equivalent to tested `1c77f200`. Signal/shared-controls #63, Paths #64 and Traffic #65 remain in review with their separately recorded validation. -[Web #65](https://github.com/MeshCore-Beacon/beacon-web/pull/65), `a928fda5b34d2663ea9a970a43bafd0406a4a133`, translates Traffic: trend/share/heatmap labels and descriptions, grouped/unassigned display labels, tooltip text, the exact IATA table and explanatory/loading/error/empty states. Exact parent is #64 at `3f2a36361cefb97751239aa1b82730a2b2585f0a`. **Merge #63, then #64, then #65.** The helper reuses validated trees without rewriting branches. +[Web #66](https://github.com/MeshCore-Beacon/beacon-web/pull/66), `8d81bd5de5ce383533e7625d04b0a05318d92606`, translates Scopes: search/metric labels, chart descriptions/remainder text, exact table, visible/total count, membership guidance and loading/error/empty/no-match states. Exact parent is #65 at `a928fda5b34d2663ea9a970a43bafd0406a4a133`. **Merge #63, #64, #65, then #66.** The helper preserves this sequence and reuses validated trees. -All **828 tests in 94 files**, build and lint pass on Windows and the native Pi; exact-head CI passes (existing CodeQL is skipped). Actual IATA codes, model identifiers, reception counts, grouping/sorting, recorded zeroes versus gaps, UTC/partial-hour behavior, query keys and canonical links stay unchanged. Tooltip plural selection uses the raw count while showing the existing formatted number. A language-only switch makes no additional data request. +All **836 tests in 94 files**, build and lint pass on Windows and the native Pi; exact-head CI passes (existing CodeQL is skipped). Scope names, search text/matching, metric-specific ranking/colours, counts, membership/default-node semantics, regional cache keys and number formatting stay unchanged. Switching language preserves an active search and adds no data request or time-window key. -Public French stays EN DIRECT; all 27 displayed IATA rows and values match across language switching. Desktop/320px charts and the exact table fit, with spaced numeric columns and no page overflow. Local browser checks cover the French heatmap tooltip, and public `7 j` retains `range=7d`. Source/JS/CSS hashes match, both feeds are connected and all 23 containers are preserved. Initial JS grows by 863 gzip bytes over #64 with no dependency or translation request added. +Populated browser checks use 16 controlled local scopes, including a long name and zero values. French charts, remainder labels, exact table and searches pass at desktop/320px. The real preview has **no scope records**; its French empty/no-match states stay EN DIRECT, retain search across language switches and fit at 320px. No fixture rows were inserted into the Pi. Source/JS/CSS hashes match, both feeds are connected, and all 23 containers are preserved. Initial JS grows by 1,199 gzip bytes over #65 without a new dependency. -Current Pi: server `c02317a4` / web `a928fda5`. Immediate frontend rollback is `3f2a3636`; source and distributions for both remain available, and completed staging was retired. The [preview changelog/source](https://canadaverse.org/beacon-dev/source.html) identifies the build. Other pages/dialogs and locale-aware data formatting remain under #12; physical Safari is untested. These translation slices add no analytics endpoint/page and leave the original owner-managed release freeze separate. +Current Pi: server `c02317a4` / web `8d81bd5d`. Immediate frontend rollback is `a928fda5`; source/distributions for both remain, and completed staging was retired. The [changelog/source](https://canadaverse.org/beacon-dev/source.html) identifies this build and its scope-data limitation. Other pages/dialogs and locale-aware formatting remain under #12; physical Safari is untested. The original owner-managed release freeze remains separate from these translation contributions. ## Delivered foundations @@ -80,14 +80,14 @@ The September 20 review correction moves both new aggregate APIs onto materializ September 20 validation covered native Go/PostgreSQL/HTTP behavior and **786 web tests**, plus private backup compatibility, feature-only startup failure, TLS/password files, cancellation and schema/data/sequence restoration. That review update passed 390/1280px browser checks, with ten distinct glyphs and complete-hour text; earlier chart checks also covered 320/768px. This is historical evidence for the unchanged feature code. Current revisions and corresponding-source archives are on the preview's changelog page. -The September 24 consolidation check built accepted server `c02317a4` and retained web `42ba5fcb`, identical in source to accepted `0f0a6ca5`. Its native PostgreSQL and public/browser evidence remains in the release checklist. The Pi frontend now includes #63/#64/#65 as documented above; #64 is retained for immediate rollback. The three-hour retained analytics sample does not establish 7/30-day history or production capacity. +The September 24 consolidation check built accepted server `c02317a4` and retained web `42ba5fcb`, identical in source to accepted `0f0a6ca5`. Its native PostgreSQL and public/browser evidence remains in the release checklist. The Pi frontend now includes #63/#64/#65/#66 as documented above; #65 is retained for immediate rollback. The three-hour retained analytics sample does not establish 7/30-day history or production capacity. ## Next phases The September 20 #116 investigation has a new [current-build result](https://github.com/MeshCore-Beacon/beacon-server/issues/116#issuecomment-5753626821): a 600-second unmodified Pi capture kept both feeds connected and retained 2,169 new observations, with no ping timeout, disconnect, deadline, SQLSTATE error or HTTP 5xx response. App/PostgreSQL CPU averaged 2.14%/3.96% of one core. The preceding 3h39 log likewise has no MQTT loss or deadline error. Timestamp warnings were classified separately. This did not measure callback or pool-acquisition duration and does not establish the original cause or production capacity. No application, ordering, acknowledgement or service change was made; #116 remains open. Further capture should follow a recurrence or meaningful workload change, rather than repeatedly sampling the same healthy state. 1. **Publish the coordinated consolidation releases.** Accepted-dev native/Pi/public validation and the changelog/source handoff are complete. Maintainers choose versions, create signed release commits/tags, promote main and verify release artifacts. Follow [RELEASE-CHECKLIST.md](RELEASE-CHECKLIST.md). Pause new analytics until this breakpoint; this release does not claim complete CoreScope parity. -2. **Continue listed issue #12 in small screen groups.** The English/French foundation #62 is merged; RF / Signal (#63), Paths & Hashes (#64) and Traffic (#65) are ready for review. Translate Scopes next, preserving identifiers and measurement semantics. Other page, chart and dialog text remains; keep partial coverage explicit. Refresh all open issues/reviews before choosing the next group. +2. **Continue listed issue #12 in small screen groups.** The English/French foundation #62 is merged; RF / Signal (#63), Paths & Hashes (#64), Traffic (#65) and Scopes (#66) are ready for review. Translate Talkers next, preserving identifiers and measurement semantics. Other page, chart and dialog text remains; keep partial coverage explicit. Refresh all open issues/reviews before choosing the next group. 3. **After the consolidation release: Channel Activity analytics.** Inspect existing aggregates and add bounded channel traffic/trend views. Define messages versus receptions, unknown/encrypted-channel coverage and the behavior when keys are unavailable. Keep keys and message contents out of aggregate responses; avoid raw-message paging to build statistics. 4. **Observed ambiguity and topology.** Separate static prefix conflicts from observed unresolved/ambiguous paths. Add justified route-pattern, neighbour, hop and distance views with clear provenance and bounded work. 5. **Administration and backup slices.** Non-destructive archive validation is merged in #160. Resolve browser login/session and restore authorization before an import endpoint; keep deployment-file coverage and remote/scheduled backup separate. Test restoration against disposable databases. These follow-ups do not delay the current release. @@ -101,7 +101,7 @@ The September 20 #116 investigation has a new [current-build result](https://git | [Server #99](https://github.com/MeshCore-Beacon/beacon-server/issues/99) | Advert names and ACK/TRACE/PING references are accepted; define any remaining packet-type formats | | [Server #60](https://github.com/MeshCore-Beacon/beacon-server/issues/60) | Remaining administration/worker/persistence behavior; account records do not establish login sessions | | [Server #72](https://github.com/MeshCore-Beacon/beacon-server/issues/72) | Download #154 and archive validation #160 are accepted; import, browser access, deployment-file coverage and remote/scheduled backup remain | -| [Web #12](https://github.com/MeshCore-Beacon/beacon-web/issues/12) | Foundation #62 is merged; Signal/analytics controls #63, Paths #64 and Traffic #65 are in review. Scopes and other screens, chart/dialog text and formatting remain | +| [Web #12](https://github.com/MeshCore-Beacon/beacon-web/issues/12) | Foundation #62 is merged; Signal #63, Paths #64, Traffic #65 and Scopes #66 are in review. Talkers and other screens, chart/dialog text and formatting remain | The six completed analytics/icon/map issues are closed. Refresh the five remaining issues first at every continuation; accepted partial contributions are not grounds to close broader issues. Use closing references only when a PR completes the issue's accepted scope; use related references for partial work. From c43e7763a47a549ea0a98fc20879b465b7ffd65b Mon Sep 17 00:00:00 2001 From: n30nex Date: Fri, 25 Sep 2026 03:36:37 -0400 Subject: [PATCH 17/69] docs(roadmap): prioritize independent Talkers correctness fix --- RELEASE-CHECKLIST.md | 8 ++++---- ROADMAP.md | 25 ++++++++++++++----------- 2 files changed, 18 insertions(+), 15 deletions(-) diff --git a/RELEASE-CHECKLIST.md b/RELEASE-CHECKLIST.md index 6b84d71..eedaafe 100644 --- a/RELEASE-CHECKLIST.md +++ b/RELEASE-CHECKLIST.md @@ -26,7 +26,7 @@ The web source tree is identical to tested preview `42ba5fcb`; preserve that art - [x] Refresh #55/#57 after the accepted #52/#53 squash. Their source trees were identical after the September 20 refresh; that history-only update needs no replacement Pi artifact. - [x] All eight application PRs pass their required checks on the published heads. Native PostgreSQL tests ran. The upstream web CodeQL job remains skipped under its existing policy and is not counted as a scan. -All six server PRs and four web PRs have merged. Server #156/#158 and web #54/#56/#58/#60 are closed. Docs #5 remains open; language foundation #62 has since merged, and translations #63/#64/#65/#66 remain outside this original frozen batch. The helper retired accepted parents/overlays without rebasing or force-pushing any application branch. Future work starts from freshly fetched dev. Backend endpoints still need deployment before dependent pages on each operator's target; Pi evidence is not evidence of another deployment. +All six server PRs and four web PRs have merged. Server #156/#158 and web #54/#56/#58/#60 are closed. Docs #5 remains open; language foundation #62 has since merged, and translations #63/#64/#65/#66 and independent bug fix #68 remain outside this original frozen batch. The helper retired accepted parents/overlays without rebasing or force-pushing any application branch. Future work starts from freshly fetched dev. Backend endpoints still need deployment before dependent pages on each operator's target; Pi evidence is not evidence of another deployment. ## Storage and retention boundary @@ -36,7 +36,7 @@ The stated replacement direction is lz4 compression, batched deletes, per-table A consolidation release must document its actual retention behavior and capacity limits. A future production parity cutover also needs verified durable history coverage and recovery copies. Do not infer either from example configuration or the Pi's short history. -Language foundation #62 has since merged as accepted web dev `6d3edbb6`, source-equivalent to tested build `1c77f200`. The live preview now includes Signal #63, Paths #64, Traffic #65 and Scopes #66 at `8d81bd5d`, with 836 passing native/Windows tests and public browser/source verification. The real scope dataset is empty; populated Scopes checks use a local fixture. Immediate frontend rollback is `a928fda5`; merge #63, #64, #65, then #66. The original release freeze below remains separately recorded; maintainers choose whether to advance it to include these later contributions. +Language foundation #62 has since merged as accepted web dev `6d3edbb6`, source-equivalent to tested build `1c77f200`. The live preview combines translations #63–#66 with independent Talkers fix #68 at `edc32842`: native Pi build/lint and 842 tests pass, with public/browser/source checks. Independent #68 is based directly on dev and passes 808 Windows tests plus CI; it can merge before the translation sequence and closes #67. Immediate frontend rollback is `8d81bd5d`. The real scope dataset is empty; populated Scopes checks used a local fixture. The original release freeze below remains separately recorded; maintainers choose whether to advance it to include these later contributions. ## Accepted-dev verification - 24 September @@ -45,7 +45,7 @@ Language foundation #62 has since merged as accepted web dev `6d3edbb6`, source- - [x] Accepted server is running on the preview. Source/asset hashes match; both feeds advance. Signal/Paths reconcile with SQL for global/regional 1/7/30-day selections, at 2-19 ms origin latency. Only three complete hours are populated; this does not prove 7/30-day history coverage. - [x] Browser Signal/Paths charts and map load with LIVE status and no captured warnings/errors. Unchanged frontend assets keep their actual `42ba5fcb` build/source identity and prior 786-test evidence; accepted `0f0a6ca5` has the identical tree. - [x] Only the Beacon app restarted; the other 22 containers and configuration/migration journal were preserved. Immediate rollback is server `5848d200` with unchanged web. The separate verifier retains its real `262eae96` binary/source identity. -- [x] Accepted review queues/overlays were retired. Start created the independent language branch directly from accepted dev; #63 followed by #64, #65 and #66 is now the active web queue. Completed issues are closed; five broader issues remain open. +- [x] Accepted review queues/overlays were retired. Start created the independent language branch directly from accepted dev; #63 followed by #64, #65 and #66 is the translation queue, with #68 independently included in preview checks. Completed issues are closed; five broader issues and focused #67 remain open. - [ ] Maintainer version selection, signed release commits, main promotion, tags and tag-built artifact verification. Stable releases remain v1.6.0 / v1.3.0. ## Earlier combined-candidate evidence - 20 September @@ -58,7 +58,7 @@ Language foundation #62 has since merged as accepted web dev `6d3edbb6`, source- - [x] Real preview analytics reconcile with SQL for the materialized window. Browser charts, complete-hour text, small screens and error/empty/retry states are verified. Both MQTT feeds advance and the public browser reports LIVE. - [x] Current and rollback server/web artifacts, exact source offers and visible changelog match the running pair. Only the Beacon app restarted; the other 20 containers were preserved. Additive rollup migrations retain observations and are compatible with the previous binary. -The current immediate rollback restores frontend `a928fda5` with server `c02317a4`. Restore accepted frontend `42ba5fcb` before using the older consolidation server rollback to `5848d200`; its metadata describes the accepted frontend. The September 20 packet-reference rollback to `6be0f762` is an older recovery point. Exact artifacts/runners are retained; application rollback keeps additive rollup views and does not remove history. +The current immediate rollback restores frontend `8d81bd5d` with server `c02317a4`. Restore accepted frontend `42ba5fcb` before using the older consolidation server rollback to `5848d200`; its metadata describes the accepted frontend. The September 20 packet-reference rollback to `6be0f762` is an older recovery point. Exact artifacts/runners are retained; application rollback keeps additive rollup views and does not remove history. ## Maintainer release handoff diff --git a/ROADMAP.md b/ROADMAP.md index 3f008cb..d54f7ee 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -25,7 +25,7 @@ Current sites: ## Accepted consolidation batch -All ten server/web contributions merged into `dev` on 24 September UTC. The accepted batch has no remaining rebase dependencies. Language foundation [web #62](https://github.com/MeshCore-Beacon/beacon-web/pull/62) has also merged as `6d3edbb6`. Signal translation [web #63](https://github.com/MeshCore-Beacon/beacon-web/pull/63), Paths translation [web #64](https://github.com/MeshCore-Beacon/beacon-web/pull/64), Traffic translation [web #65](https://github.com/MeshCore-Beacon/beacon-web/pull/65), Scopes translation [web #66](https://github.com/MeshCore-Beacon/beacon-web/pull/66), and [docs #5](https://github.com/MeshCore-Beacon/beacon-docs/pull/5) remain open. +All ten server/web contributions merged into `dev` on 24 September UTC. The accepted batch has no remaining rebase dependencies. Language foundation [web #62](https://github.com/MeshCore-Beacon/beacon-web/pull/62) has also merged as `6d3edbb6`. Signal translation [web #63](https://github.com/MeshCore-Beacon/beacon-web/pull/63), Paths translation [web #64](https://github.com/MeshCore-Beacon/beacon-web/pull/64), Traffic translation [web #65](https://github.com/MeshCore-Beacon/beacon-web/pull/65), Scopes translation [web #66](https://github.com/MeshCore-Beacon/beacon-web/pull/66), independent Talkers fix [web #68](https://github.com/MeshCore-Beacon/beacon-web/pull/68), and [docs #5](https://github.com/MeshCore-Beacon/beacon-docs/pull/5) remain open. The original consolidation freeze is server `c02317a4ac7228d19cab498edfa1d61186c84626` / web `0f0a6ca51c7b2c3315db77954f61b30bbdeea5e2`. CI and image builds pass at those heads; web CodeQL remains skipped and is not a security scan. Current web dev has since advanced to `6d3edbb6` through #62. Stable releases are still server **v1.6.0** and web **v1.3.0**. Merged dev is not a published stable release. @@ -42,21 +42,23 @@ The original consolidation freeze is server `c02317a4ac7228d19cab498edfa1d61186c | [Web #59](https://github.com/MeshCore-Beacon/beacon-web/pull/59) | distinguish analytics navigation icons | `c2ab29a5` | | [Web #61](https://github.com/MeshCore-Beacon/beacon-web/pull/61) | omit reset and invalid node locations | `0f0a6ca5` | -Server #156/#158 and web #54/#56/#58/#60 are closed. Server #60/#72/#99/#116 and web #12 remain open: the accepted work completes slices, not their remaining scope. Archive verification establishes structure and integrity, not authenticity, SQL safety or restorability. Packet-carried ACK/TRACE/PING references are not identity or delivery guarantees. +Server #156/#158 and web #54/#56/#58/#60 are closed. Server #60/#72/#99/#116 and web #12 remain open for their remaining scope. New web #67 has a complete fix in independent #68 awaiting acceptance. Archive verification establishes structure and integrity, not authenticity, SQL safety or restorability. Packet-carried ACK/TRACE/PING references are not identity or delivery guarantees. The workflow removed the accepted entries and overlays without rebasing or force-pushing any application branch. Signal translation #63 started directly from fresh dev `6d3edbb6`; Paths translation #64 follows its exact `01c6d1aa` head. Traffic translation #65 follows #64 at `3f2a3636`. Scopes #66 follows #65 at `a928fda5`. Merge #63, #64, #65, then #66. Future overlapping follow-ups use the declared parent through the helper. Preserve actual build identities when accepted commits have identical source trees. See [the contributor workflow](CONTRIBUTOR_WORKFLOW.md) and [consolidation checklist](RELEASE-CHECKLIST.md). ## Current independent issue work -Language foundation #62 is merged as accepted dev `6d3edbb61bb79fe7aa336c5ae93a6c0a3f6f7545`, source-equivalent to tested `1c77f200`. Signal/shared-controls #63, Paths #64 and Traffic #65 remain in review with their separately recorded validation. +**Correctness before translation:** [issue #67](https://github.com/MeshCore-Beacon/beacon-web/issues/67) was reproduced while inspecting Talkers. Old results remained visible during filter changes, and cached advertiser rows survived a failed refresh. [PR #68](https://github.com/MeshCore-Beacon/beacon-web/pull/68), `2e3b5161514e4d5209133a9db32f311a6d756dcf`, fixes those per-panel states while preserving valid background refreshes and healthy independent panels. Its closing reference targets #67; the issue remains open until acceptance. -[Web #66](https://github.com/MeshCore-Beacon/beacon-web/pull/66), `8d81bd5de5ce383533e7625d04b0a05318d92606`, translates Scopes: search/metric labels, chart descriptions/remainder text, exact table, visible/total count, membership guidance and loading/error/empty/no-match states. Exact parent is #65 at `a928fda5b34d2663ea9a970a43bafd0406a4a133`. **Merge #63, #64, #65, then #66.** The helper preserves this sequence and reuses validated trees. +**#68 can merge independently**, directly into current dev `6d3edbb6`; it does not wait for translations #63–#66. The independent source passes Windows build/lint, **808 tests in 94 files**, and exact-head CI. Three regressions failed first; six real QueryClient tests prove delayed filter transitions, cached-error recovery, independent panels and normal background refresh behavior. APIs, query keys, retry policy, counts/rates, sorting and retention stay unchanged. -All **836 tests in 94 files**, build and lint pass on Windows and the native Pi; exact-head CI passes (existing CodeQL is skipped). Scope names, search text/matching, metric-specific ranking/colours, counts, membership/default-node semantics, regional cache keys and number formatting stay unchanged. Switching language preserves an active search and adds no data request or time-window key. +The main preview keeps the translation sequence #63 -> #64 -> #65 -> #66 and includes #68 as an independent input. Its actual source is `edc32842c189855fdd6757c4ba1538b2c4ccd10a`, combining #66 `8d81bd5d` and fix `2e3b5161`. Native Pi build/lint and **842 tests in 95 files** pass. All five application PR heads check green under their existing policy (CodeQL is skipped); the helper reuses validated trees without rewriting existing branches. -Populated browser checks use 16 controlled local scopes, including a long name and zero values. French charts, remainder labels, exact table and searches pass at desktop/320px. The real preview has **no scope records**; its French empty/no-match states stay EN DIRECT, retain search across language switches and fit at 320px. No fixture rows were inserted into the Pi. Source/JS/CSS hashes match, both feeds are connected, and all 23 containers are preserved. Initial JS grows by 1,199 gzip bytes over #65 without a new dependency. +Controlled local browser faults confirm loading/error/recovery behavior and a usable healthy panel. The public preview stays LIVE through range changes and fits at 320px, with 20 advertiser rows and a valid empty sender panel. No failures or test records were injected into the Pi. Source/JS/CSS hashes match, both brokers are connected, and all 23 containers remain unchanged. Initial JS grows by 2 gzip bytes over the preceding preview. -Current Pi: server `c02317a4` / web `8d81bd5d`. Immediate frontend rollback is `a928fda5`; source/distributions for both remain, and completed staging was retired. The [changelog/source](https://canadaverse.org/beacon-dev/source.html) identifies this build and its scope-data limitation. Other pages/dialogs and locale-aware formatting remain under #12; physical Safari is untested. The original owner-managed release freeze remains separate from these translation contributions. +Current Pi: server `c02317a4` / web `edc32842`. Immediate frontend rollback is `8d81bd5d`; current/rollback artifacts and source remain, and completed staging was retired. The [source/changelog](https://canadaverse.org/beacon-dev/source.html) distinguishes the independent PR from the combined preview. The real scope dataset is still empty; prior populated Scopes proof uses local data. Physical Safari, durable history and production capacity remain unverified release gates. + +Language foundation #62 is merged; #63–#66 remain in review under #12. Talkers translation follows acceptance of #68, including stable table column IDs to preserve sorting across languages. Clock Drift translation can proceed independently while #68 is reviewed. Keep the bug-fix input when composing later previews. The original consolidation release freeze remains separately owned by the maintainers. ## Delivered foundations @@ -80,14 +82,14 @@ The September 20 review correction moves both new aggregate APIs onto materializ September 20 validation covered native Go/PostgreSQL/HTTP behavior and **786 web tests**, plus private backup compatibility, feature-only startup failure, TLS/password files, cancellation and schema/data/sequence restoration. That review update passed 390/1280px browser checks, with ten distinct glyphs and complete-hour text; earlier chart checks also covered 320/768px. This is historical evidence for the unchanged feature code. Current revisions and corresponding-source archives are on the preview's changelog page. -The September 24 consolidation check built accepted server `c02317a4` and retained web `42ba5fcb`, identical in source to accepted `0f0a6ca5`. Its native PostgreSQL and public/browser evidence remains in the release checklist. The Pi frontend now includes #63/#64/#65/#66 as documented above; #65 is retained for immediate rollback. The three-hour retained analytics sample does not establish 7/30-day history or production capacity. +The September 24 consolidation check built accepted server `c02317a4` and retained web `42ba5fcb`, identical in source to accepted `0f0a6ca5`. Its native PostgreSQL and public/browser evidence remains in the release checklist. The Pi frontend includes #63–#66 plus independent #68 as documented above; #66 is retained for immediate rollback. The three-hour retained analytics sample does not establish 7/30-day history or production capacity. ## Next phases The September 20 #116 investigation has a new [current-build result](https://github.com/MeshCore-Beacon/beacon-server/issues/116#issuecomment-5753626821): a 600-second unmodified Pi capture kept both feeds connected and retained 2,169 new observations, with no ping timeout, disconnect, deadline, SQLSTATE error or HTTP 5xx response. App/PostgreSQL CPU averaged 2.14%/3.96% of one core. The preceding 3h39 log likewise has no MQTT loss or deadline error. Timestamp warnings were classified separately. This did not measure callback or pool-acquisition duration and does not establish the original cause or production capacity. No application, ordering, acknowledgement or service change was made; #116 remains open. Further capture should follow a recurrence or meaningful workload change, rather than repeatedly sampling the same healthy state. 1. **Publish the coordinated consolidation releases.** Accepted-dev native/Pi/public validation and the changelog/source handoff are complete. Maintainers choose versions, create signed release commits/tags, promote main and verify release artifacts. Follow [RELEASE-CHECKLIST.md](RELEASE-CHECKLIST.md). Pause new analytics until this breakpoint; this release does not claim complete CoreScope parity. -2. **Continue listed issue #12 in small screen groups.** The English/French foundation #62 is merged; RF / Signal (#63), Paths & Hashes (#64), Traffic (#65) and Scopes (#66) are ready for review. Translate Talkers next, preserving identifiers and measurement semantics. Other page, chart and dialog text remains; keep partial coverage explicit. Refresh all open issues/reviews before choosing the next group. +2. **Continue listed issue #12 in small screen groups.** The English/French foundation #62 is merged; RF / Signal (#63), Paths & Hashes (#64), Traffic (#65) and Scopes (#66) are ready for review. Prioritize the independent #67 fix in #68. Translate Talkers after it lands; Clock Drift is independent work while reviewed. Preserve identifiers and measurement semantics. Other page, chart and dialog text remains; keep partial coverage explicit. Refresh all open issues/reviews before choosing the next group. 3. **After the consolidation release: Channel Activity analytics.** Inspect existing aggregates and add bounded channel traffic/trend views. Define messages versus receptions, unknown/encrypted-channel coverage and the behavior when keys are unavailable. Keep keys and message contents out of aggregate responses; avoid raw-message paging to build statistics. 4. **Observed ambiguity and topology.** Separate static prefix conflicts from observed unresolved/ambiguous paths. Add justified route-pattern, neighbour, hop and distance views with clear provenance and bounded work. 5. **Administration and backup slices.** Non-destructive archive validation is merged in #160. Resolve browser login/session and restore authorization before an import endpoint; keep deployment-file coverage and remote/scheduled backup separate. Test restoration against disposable databases. These follow-ups do not delay the current release. @@ -101,9 +103,10 @@ The September 20 #116 investigation has a new [current-build result](https://git | [Server #99](https://github.com/MeshCore-Beacon/beacon-server/issues/99) | Advert names and ACK/TRACE/PING references are accepted; define any remaining packet-type formats | | [Server #60](https://github.com/MeshCore-Beacon/beacon-server/issues/60) | Remaining administration/worker/persistence behavior; account records do not establish login sessions | | [Server #72](https://github.com/MeshCore-Beacon/beacon-server/issues/72) | Download #154 and archive validation #160 are accepted; import, browser access, deployment-file coverage and remote/scheduled backup remain | -| [Web #12](https://github.com/MeshCore-Beacon/beacon-web/issues/12) | Foundation #62 is merged; Signal #63, Paths #64, Traffic #65 and Scopes #66 are in review. Talkers and other screens, chart/dialog text and formatting remain | +| [Web #67](https://github.com/MeshCore-Beacon/beacon-web/issues/67) | Stale Talkers display states are fixed in independent #68; closes on merge into dev | +| [Web #12](https://github.com/MeshCore-Beacon/beacon-web/issues/12) | Foundation #62 is merged; Signal #63, Paths #64, Traffic #65 and Scopes #66 are in review. Talkers follows #68; Clock Drift and other screens, chart/dialog text and formatting remain | -The six completed analytics/icon/map issues are closed. Refresh the five remaining issues first at every continuation; accepted partial contributions are not grounds to close broader issues. Use closing references only when a PR completes the issue's accepted scope; use related references for partial work. +The six completed analytics/icon/map issues are closed. Refresh the six currently open issues first at every continuation; accepted partial contributions are not grounds to close broader issues. Use closing references only when a PR completes the issue's accepted scope; use related references for partial work. ## Production parity matrix From 8936160a008270e7cecf0e7e8c9d15596dcb3c3f Mon Sep 17 00:00:00 2001 From: n30nex Date: Fri, 25 Sep 2026 05:07:31 -0400 Subject: [PATCH 18/69] docs(roadmap): record Clock Drift translation and stable sorting --- RELEASE-CHECKLIST.md | 8 ++++---- ROADMAP.md | 24 ++++++++++++------------ 2 files changed, 16 insertions(+), 16 deletions(-) diff --git a/RELEASE-CHECKLIST.md b/RELEASE-CHECKLIST.md index eedaafe..0d9efa3 100644 --- a/RELEASE-CHECKLIST.md +++ b/RELEASE-CHECKLIST.md @@ -26,7 +26,7 @@ The web source tree is identical to tested preview `42ba5fcb`; preserve that art - [x] Refresh #55/#57 after the accepted #52/#53 squash. Their source trees were identical after the September 20 refresh; that history-only update needs no replacement Pi artifact. - [x] All eight application PRs pass their required checks on the published heads. Native PostgreSQL tests ran. The upstream web CodeQL job remains skipped under its existing policy and is not counted as a scan. -All six server PRs and four web PRs have merged. Server #156/#158 and web #54/#56/#58/#60 are closed. Docs #5 remains open; language foundation #62 has since merged, and translations #63/#64/#65/#66 and independent bug fix #68 remain outside this original frozen batch. The helper retired accepted parents/overlays without rebasing or force-pushing any application branch. Future work starts from freshly fetched dev. Backend endpoints still need deployment before dependent pages on each operator's target; Pi evidence is not evidence of another deployment. +All six server PRs and four web PRs have merged. Server #156/#158 and web #54/#56/#58/#60 are closed. Docs #5 remains open; language foundation #62 has since merged, and translations #63/#64/#65/#66/#69 and independent bug fix #68 remain outside this original frozen batch. The helper retired accepted parents/overlays without rebasing or force-pushing any application branch. Future work starts from freshly fetched dev. Backend endpoints still need deployment before dependent pages on each operator's target; Pi evidence is not evidence of another deployment. ## Storage and retention boundary @@ -36,7 +36,7 @@ The stated replacement direction is lz4 compression, batched deletes, per-table A consolidation release must document its actual retention behavior and capacity limits. A future production parity cutover also needs verified durable history coverage and recovery copies. Do not infer either from example configuration or the Pi's short history. -Language foundation #62 has since merged as accepted web dev `6d3edbb6`, source-equivalent to tested build `1c77f200`. The live preview combines translations #63–#66 with independent Talkers fix #68 at `edc32842`: native Pi build/lint and 842 tests pass, with public/browser/source checks. Independent #68 is based directly on dev and passes 808 Windows tests plus CI; it can merge before the translation sequence and closes #67. Immediate frontend rollback is `8d81bd5d`. The real scope dataset is empty; populated Scopes checks used a local fixture. The original release freeze below remains separately recorded; maintainers choose whether to advance it to include these later contributions. +Language foundation #62 has since merged as accepted web dev `6d3edbb6`, source-equivalent to tested build `1c77f200`. The live preview combines translation #69 (following #63–#66) with independent #68 at `8fb636aa`: native Pi build/lint and 853 tests pass, with public/browser/source checks. The Clock Drift PR source passes 847 Windows tests/CI; independent #68 retains its 808-test evidence and can merge first to close #67. Immediate frontend rollback is combined `edc32842`. Shared Timestamp wording remains separate; the real scope dataset is empty. The original release freeze below remains separately recorded; maintainers choose whether to advance it to include these later contributions. ## Accepted-dev verification - 24 September @@ -45,7 +45,7 @@ Language foundation #62 has since merged as accepted web dev `6d3edbb6`, source- - [x] Accepted server is running on the preview. Source/asset hashes match; both feeds advance. Signal/Paths reconcile with SQL for global/regional 1/7/30-day selections, at 2-19 ms origin latency. Only three complete hours are populated; this does not prove 7/30-day history coverage. - [x] Browser Signal/Paths charts and map load with LIVE status and no captured warnings/errors. Unchanged frontend assets keep their actual `42ba5fcb` build/source identity and prior 786-test evidence; accepted `0f0a6ca5` has the identical tree. - [x] Only the Beacon app restarted; the other 22 containers and configuration/migration journal were preserved. Immediate rollback is server `5848d200` with unchanged web. The separate verifier retains its real `262eae96` binary/source identity. -- [x] Accepted review queues/overlays were retired. Start created the independent language branch directly from accepted dev; #63 followed by #64, #65 and #66 is the translation queue, with #68 independently included in preview checks. Completed issues are closed; five broader issues and focused #67 remain open. +- [x] Accepted review queues/overlays were retired. Start created the independent language branch directly from accepted dev; #63 followed by #64, #65, #66 and #69 is the translation queue, with #68 independently included in preview checks. Completed issues are closed; five broader issues and focused #67 remain open. - [ ] Maintainer version selection, signed release commits, main promotion, tags and tag-built artifact verification. Stable releases remain v1.6.0 / v1.3.0. ## Earlier combined-candidate evidence - 20 September @@ -58,7 +58,7 @@ Language foundation #62 has since merged as accepted web dev `6d3edbb6`, source- - [x] Real preview analytics reconcile with SQL for the materialized window. Browser charts, complete-hour text, small screens and error/empty/retry states are verified. Both MQTT feeds advance and the public browser reports LIVE. - [x] Current and rollback server/web artifacts, exact source offers and visible changelog match the running pair. Only the Beacon app restarted; the other 20 containers were preserved. Additive rollup migrations retain observations and are compatible with the previous binary. -The current immediate rollback restores frontend `8d81bd5d` with server `c02317a4`. Restore accepted frontend `42ba5fcb` before using the older consolidation server rollback to `5848d200`; its metadata describes the accepted frontend. The September 20 packet-reference rollback to `6be0f762` is an older recovery point. Exact artifacts/runners are retained; application rollback keeps additive rollup views and does not remove history. +The current immediate rollback restores combined frontend `edc32842` with server `c02317a4`. Restore accepted frontend `42ba5fcb` before using the older consolidation server rollback to `5848d200`; its metadata describes the accepted frontend. The September 20 packet-reference rollback to `6be0f762` is an older recovery point. Exact artifacts/runners are retained; application rollback keeps additive rollup views and does not remove history. ## Maintainer release handoff diff --git a/ROADMAP.md b/ROADMAP.md index d54f7ee..8063f8f 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -25,7 +25,7 @@ Current sites: ## Accepted consolidation batch -All ten server/web contributions merged into `dev` on 24 September UTC. The accepted batch has no remaining rebase dependencies. Language foundation [web #62](https://github.com/MeshCore-Beacon/beacon-web/pull/62) has also merged as `6d3edbb6`. Signal translation [web #63](https://github.com/MeshCore-Beacon/beacon-web/pull/63), Paths translation [web #64](https://github.com/MeshCore-Beacon/beacon-web/pull/64), Traffic translation [web #65](https://github.com/MeshCore-Beacon/beacon-web/pull/65), Scopes translation [web #66](https://github.com/MeshCore-Beacon/beacon-web/pull/66), independent Talkers fix [web #68](https://github.com/MeshCore-Beacon/beacon-web/pull/68), and [docs #5](https://github.com/MeshCore-Beacon/beacon-docs/pull/5) remain open. +All ten server/web contributions merged into `dev` on 24 September UTC. The accepted batch has no remaining rebase dependencies. Language foundation [web #62](https://github.com/MeshCore-Beacon/beacon-web/pull/62) has also merged as `6d3edbb6`. Signal translation [web #63](https://github.com/MeshCore-Beacon/beacon-web/pull/63), Paths translation [web #64](https://github.com/MeshCore-Beacon/beacon-web/pull/64), Traffic translation [web #65](https://github.com/MeshCore-Beacon/beacon-web/pull/65), Scopes translation [web #66](https://github.com/MeshCore-Beacon/beacon-web/pull/66), Clock Drift translation [web #69](https://github.com/MeshCore-Beacon/beacon-web/pull/69), independent Talkers fix [web #68](https://github.com/MeshCore-Beacon/beacon-web/pull/68), and [docs #5](https://github.com/MeshCore-Beacon/beacon-docs/pull/5) remain open. The original consolidation freeze is server `c02317a4ac7228d19cab498edfa1d61186c84626` / web `0f0a6ca51c7b2c3315db77954f61b30bbdeea5e2`. CI and image builds pass at those heads; web CodeQL remains skipped and is not a security scan. Current web dev has since advanced to `6d3edbb6` through #62. Stable releases are still server **v1.6.0** and web **v1.3.0**. Merged dev is not a published stable release. @@ -44,21 +44,21 @@ The original consolidation freeze is server `c02317a4ac7228d19cab498edfa1d61186c Server #156/#158 and web #54/#56/#58/#60 are closed. Server #60/#72/#99/#116 and web #12 remain open for their remaining scope. New web #67 has a complete fix in independent #68 awaiting acceptance. Archive verification establishes structure and integrity, not authenticity, SQL safety or restorability. Packet-carried ACK/TRACE/PING references are not identity or delivery guarantees. -The workflow removed the accepted entries and overlays without rebasing or force-pushing any application branch. Signal translation #63 started directly from fresh dev `6d3edbb6`; Paths translation #64 follows its exact `01c6d1aa` head. Traffic translation #65 follows #64 at `3f2a3636`. Scopes #66 follows #65 at `a928fda5`. Merge #63, #64, #65, then #66. Future overlapping follow-ups use the declared parent through the helper. Preserve actual build identities when accepted commits have identical source trees. See [the contributor workflow](CONTRIBUTOR_WORKFLOW.md) and [consolidation checklist](RELEASE-CHECKLIST.md). +The workflow removed the accepted entries and overlays without rebasing or force-pushing any application branch. Signal translation #63 started directly from fresh dev `6d3edbb6`; Paths translation #64 follows its exact `01c6d1aa` head. Traffic translation #65 follows #64 at `3f2a3636`. Scopes #66 follows #65 at `a928fda5`. Clock Drift #69 follows #66 at `8d81bd5d`. Merge #63, #64, #65, #66, then #69. Future overlapping follow-ups use the declared parent through the helper. Preserve actual build identities when accepted commits have identical source trees. See [the contributor workflow](CONTRIBUTOR_WORKFLOW.md) and [consolidation checklist](RELEASE-CHECKLIST.md). -## Current independent issue work +## Current issue work -**Correctness before translation:** [issue #67](https://github.com/MeshCore-Beacon/beacon-web/issues/67) was reproduced while inspecting Talkers. Old results remained visible during filter changes, and cached advertiser rows survived a failed refresh. [PR #68](https://github.com/MeshCore-Beacon/beacon-web/pull/68), `2e3b5161514e4d5209133a9db32f311a6d756dcf`, fixes those per-panel states while preserving valid background refreshes and healthy independent panels. Its closing reference targets #67; the issue remains open until acceptance. +**Independent correctness fix:** [#68](https://github.com/MeshCore-Beacon/beacon-web/pull/68), `2e3b5161`, remains ready directly on dev and can merge before the translation queue. It closes focused [#67](https://github.com/MeshCore-Beacon/beacon-web/issues/67) on acceptance. The independent source's 808 Windows tests and the prior combined preview's native/browser checks stay separately recorded; Talkers translation follows that same-file fix. -**#68 can merge independently**, directly into current dev `6d3edbb6`; it does not wait for translations #63–#66. The independent source passes Windows build/lint, **808 tests in 94 files**, and exact-head CI. Three regressions failed first; six real QueryClient tests prove delayed filter transitions, cached-error recovery, independent panels and normal background refresh behavior. APIs, query keys, retry policy, counts/rates, sorting and retention stay unchanged. +[Clock Drift translation #69](https://github.com/MeshCore-Beacon/beacon-web/pull/69), `efd73757b5eae58c8e9e6b3e01f11cce4603cd85`, follows #66 `8d81bd5d`. **Translation order: #63 -> #64 -> #65 -> #66 -> #69.** It translates the caption, table headings, drift direction words and empty/error labels. Stable column IDs preserve selected sorting and focus as labels change; existing header-based callers remain compatible. Cached pending/error rows use the unavailable state, while healthy background refreshes retain valid data. -The main preview keeps the translation sequence #63 -> #64 -> #65 -> #66 and includes #68 as an independent input. Its actual source is `edc32842c189855fdd6757c4ba1538b2c4ccd10a`, combining #66 `8d81bd5d` and fix `2e3b5161`. Native Pi build/lint and **842 tests in 95 files** pass. All five application PR heads check green under their existing policy (CodeQL is skipped); the helper reuses validated trees without rewriting existing branches. +The PR source passes Windows build/lint and **847 tests in 96 files**, plus exact-head CI. The actual Pi source `8fb636aafaa303808e6e695f8459d3ac639f9589` also contains independent #68, and passes native build/lint and **853 tests in 97 files**. Signs, units, rounding, warning thresholds, identities/IATAs and regional query keys are preserved. Shared Timestamp formatting and relative “ago” wording remain a later slice. -Controlled local browser faults confirm loading/error/recovery behavior and a usable healthy panel. The public preview stays LIVE through range changes and fits at 320px, with 20 advertiser rows and a valid empty sender panel. No failures or test records were injected into the Pi. Source/JS/CSS hashes match, both brokers are connected, and all 23 containers remain unchanged. Initial JS grows by 2 gzip bytes over the preceding preview. +Local/public browser checks preserve selected node order and signed magnitudes across French switching for 100 live rows, and translated controls restore worst-first sorting. Public French stays EN DIRECT. The 320px layout contains horizontal table scrolling without page overflow. Source/JS/CSS hashes match, both brokers are connected and all 23 containers are unchanged. Initial JS grows by 246 gzip bytes over the previous combined preview. Physical Safari remains untested. -Current Pi: server `c02317a4` / web `edc32842`. Immediate frontend rollback is `8d81bd5d`; current/rollback artifacts and source remain, and completed staging was retired. The [source/changelog](https://canadaverse.org/beacon-dev/source.html) distinguishes the independent PR from the combined preview. The real scope dataset is still empty; prior populated Scopes proof uses local data. Physical Safari, durable history and production capacity remain unverified release gates. +Current Pi: server `c02317a4` / combined web `8fb636aa`. Immediate frontend rollback is `edc32842`, which retains the Talkers fix. Current/rollback artifacts and source remain; completed staging was retired. The [source/changelog](https://canadaverse.org/beacon-dev/source.html) distinguishes PR and combined-build identities. The helper retains independent #68, checks all six application PR heads and reuses validated trees without rewriting existing branches. -Language foundation #62 is merged; #63–#66 remain in review under #12. Talkers translation follows acceptance of #68, including stable table column IDs to preserve sorting across languages. Clock Drift translation can proceed independently while #68 is reviewed. Keep the bug-fix input when composing later previews. The original consolidation release freeze remains separately owned by the maintainers. +Next under #12: shared Timestamp wording while #68 is reviewed, then Talkers translation after its fix lands and one stack refresh. Reuse the stable column IDs from #69 for translated sortable tables. The original consolidation release freeze remains separately owned by the maintainers; history/capacity and physical Safari remain explicit gates. ## Delivered foundations @@ -82,14 +82,14 @@ The September 20 review correction moves both new aggregate APIs onto materializ September 20 validation covered native Go/PostgreSQL/HTTP behavior and **786 web tests**, plus private backup compatibility, feature-only startup failure, TLS/password files, cancellation and schema/data/sequence restoration. That review update passed 390/1280px browser checks, with ten distinct glyphs and complete-hour text; earlier chart checks also covered 320/768px. This is historical evidence for the unchanged feature code. Current revisions and corresponding-source archives are on the preview's changelog page. -The September 24 consolidation check built accepted server `c02317a4` and retained web `42ba5fcb`, identical in source to accepted `0f0a6ca5`. Its native PostgreSQL and public/browser evidence remains in the release checklist. The Pi frontend includes #63–#66 plus independent #68 as documented above; #66 is retained for immediate rollback. The three-hour retained analytics sample does not establish 7/30-day history or production capacity. +The September 24 consolidation check built accepted server `c02317a4` and retained web `42ba5fcb`, identical in source to accepted `0f0a6ca5`. Its native PostgreSQL and public/browser evidence remains in the release checklist. The Pi frontend includes #63–#66 and #69 plus independent #68 as documented above; the prior combined preview is retained for rollback. The three-hour retained analytics sample does not establish 7/30-day history or production capacity. ## Next phases The September 20 #116 investigation has a new [current-build result](https://github.com/MeshCore-Beacon/beacon-server/issues/116#issuecomment-5753626821): a 600-second unmodified Pi capture kept both feeds connected and retained 2,169 new observations, with no ping timeout, disconnect, deadline, SQLSTATE error or HTTP 5xx response. App/PostgreSQL CPU averaged 2.14%/3.96% of one core. The preceding 3h39 log likewise has no MQTT loss or deadline error. Timestamp warnings were classified separately. This did not measure callback or pool-acquisition duration and does not establish the original cause or production capacity. No application, ordering, acknowledgement or service change was made; #116 remains open. Further capture should follow a recurrence or meaningful workload change, rather than repeatedly sampling the same healthy state. 1. **Publish the coordinated consolidation releases.** Accepted-dev native/Pi/public validation and the changelog/source handoff are complete. Maintainers choose versions, create signed release commits/tags, promote main and verify release artifacts. Follow [RELEASE-CHECKLIST.md](RELEASE-CHECKLIST.md). Pause new analytics until this breakpoint; this release does not claim complete CoreScope parity. -2. **Continue listed issue #12 in small screen groups.** The English/French foundation #62 is merged; RF / Signal (#63), Paths & Hashes (#64), Traffic (#65) and Scopes (#66) are ready for review. Prioritize the independent #67 fix in #68. Translate Talkers after it lands; Clock Drift is independent work while reviewed. Preserve identifiers and measurement semantics. Other page, chart and dialog text remains; keep partial coverage explicit. Refresh all open issues/reviews before choosing the next group. +2. **Continue listed issue #12 in small screen groups.** The English/French foundation #62 is merged; RF / Signal (#63), Paths & Hashes (#64), Traffic (#65), Scopes (#66) and Clock Drift (#69) are ready for review. Prioritize #68 for #67. Shared Timestamp wording is next while reviewed; Talkers translation follows the fix. Preserve identifiers and measurement semantics. Other page, chart and dialog text remains; keep partial coverage explicit. Refresh all open issues/reviews before choosing the next group. 3. **After the consolidation release: Channel Activity analytics.** Inspect existing aggregates and add bounded channel traffic/trend views. Define messages versus receptions, unknown/encrypted-channel coverage and the behavior when keys are unavailable. Keep keys and message contents out of aggregate responses; avoid raw-message paging to build statistics. 4. **Observed ambiguity and topology.** Separate static prefix conflicts from observed unresolved/ambiguous paths. Add justified route-pattern, neighbour, hop and distance views with clear provenance and bounded work. 5. **Administration and backup slices.** Non-destructive archive validation is merged in #160. Resolve browser login/session and restore authorization before an import endpoint; keep deployment-file coverage and remote/scheduled backup separate. Test restoration against disposable databases. These follow-ups do not delay the current release. @@ -104,7 +104,7 @@ The September 20 #116 investigation has a new [current-build result](https://git | [Server #60](https://github.com/MeshCore-Beacon/beacon-server/issues/60) | Remaining administration/worker/persistence behavior; account records do not establish login sessions | | [Server #72](https://github.com/MeshCore-Beacon/beacon-server/issues/72) | Download #154 and archive validation #160 are accepted; import, browser access, deployment-file coverage and remote/scheduled backup remain | | [Web #67](https://github.com/MeshCore-Beacon/beacon-web/issues/67) | Stale Talkers display states are fixed in independent #68; closes on merge into dev | -| [Web #12](https://github.com/MeshCore-Beacon/beacon-web/issues/12) | Foundation #62 is merged; Signal #63, Paths #64, Traffic #65 and Scopes #66 are in review. Talkers follows #68; Clock Drift and other screens, chart/dialog text and formatting remain | +| [Web #12](https://github.com/MeshCore-Beacon/beacon-web/issues/12) | Foundation #62 is merged; Signal #63, Paths #64, Traffic #65, Scopes #66 and Clock Drift #69 are in review. Shared Timestamp wording is next; Talkers follows #68, with other screens/dialogs/formatting remaining | The six completed analytics/icon/map issues are closed. Refresh the six currently open issues first at every continuation; accepted partial contributions are not grounds to close broader issues. Use closing references only when a PR completes the issue's accepted scope; use related references for partial work. From a9acc29950081eaf427c74e9ac37db9d8bf281b4 Mon Sep 17 00:00:00 2001 From: n30nex Date: Fri, 25 Sep 2026 05:36:23 -0400 Subject: [PATCH 19/69] docs(roadmap): record shared timestamp translation --- RELEASE-CHECKLIST.md | 8 ++++---- ROADMAP.md | 24 +++++++++++++----------- 2 files changed, 17 insertions(+), 15 deletions(-) diff --git a/RELEASE-CHECKLIST.md b/RELEASE-CHECKLIST.md index 0d9efa3..ff60c16 100644 --- a/RELEASE-CHECKLIST.md +++ b/RELEASE-CHECKLIST.md @@ -26,7 +26,7 @@ The web source tree is identical to tested preview `42ba5fcb`; preserve that art - [x] Refresh #55/#57 after the accepted #52/#53 squash. Their source trees were identical after the September 20 refresh; that history-only update needs no replacement Pi artifact. - [x] All eight application PRs pass their required checks on the published heads. Native PostgreSQL tests ran. The upstream web CodeQL job remains skipped under its existing policy and is not counted as a scan. -All six server PRs and four web PRs have merged. Server #156/#158 and web #54/#56/#58/#60 are closed. Docs #5 remains open; language foundation #62 has since merged, and translations #63/#64/#65/#66/#69 and independent bug fix #68 remain outside this original frozen batch. The helper retired accepted parents/overlays without rebasing or force-pushing any application branch. Future work starts from freshly fetched dev. Backend endpoints still need deployment before dependent pages on each operator's target; Pi evidence is not evidence of another deployment. +All six server PRs and four web PRs have merged. Server #156/#158 and web #54/#56/#58/#60 are closed. Docs #5 remains open; language foundation #62 has since merged, and translations #63/#64/#65/#66/#69/#70 and independent bug fix #68 remain outside this original frozen batch. The helper retired accepted parents/overlays without rebasing or force-pushing any application branch. Future work starts from freshly fetched dev. Backend endpoints still need deployment before dependent pages on each operator's target; Pi evidence is not evidence of another deployment. ## Storage and retention boundary @@ -36,7 +36,7 @@ The stated replacement direction is lz4 compression, batched deletes, per-table A consolidation release must document its actual retention behavior and capacity limits. A future production parity cutover also needs verified durable history coverage and recovery copies. Do not infer either from example configuration or the Pi's short history. -Language foundation #62 has since merged as accepted web dev `6d3edbb6`, source-equivalent to tested build `1c77f200`. The live preview combines translation #69 (following #63–#66) with independent #68 at `8fb636aa`: native Pi build/lint and 853 tests pass, with public/browser/source checks. The Clock Drift PR source passes 847 Windows tests/CI; independent #68 retains its 808-test evidence and can merge first to close #67. Immediate frontend rollback is combined `edc32842`. Shared Timestamp wording remains separate; the real scope dataset is empty. The original release freeze below remains separately recorded; maintainers choose whether to advance it to include these later contributions. +Language foundation #62 has since merged as accepted web dev `6d3edbb6`, source-equivalent to tested build `1c77f200`. The live preview combines translation #70 (following #63–#66/#69) with independent #68 at `300ee974`: native Pi build/lint and 860 tests pass, with public/browser/source checks. The Timestamp PR source passes 854 Windows tests/CI; independent #68 retains its 808-test evidence and can merge first to close #67. Immediate frontend rollback is combined `8fb636aa`. Direct per-page time phrases and broader formatting remain follow-ups; the real scope dataset is empty. The original release freeze below remains separately recorded; maintainers choose whether to advance it to include these later contributions. ## Accepted-dev verification - 24 September @@ -45,7 +45,7 @@ Language foundation #62 has since merged as accepted web dev `6d3edbb6`, source- - [x] Accepted server is running on the preview. Source/asset hashes match; both feeds advance. Signal/Paths reconcile with SQL for global/regional 1/7/30-day selections, at 2-19 ms origin latency. Only three complete hours are populated; this does not prove 7/30-day history coverage. - [x] Browser Signal/Paths charts and map load with LIVE status and no captured warnings/errors. Unchanged frontend assets keep their actual `42ba5fcb` build/source identity and prior 786-test evidence; accepted `0f0a6ca5` has the identical tree. - [x] Only the Beacon app restarted; the other 22 containers and configuration/migration journal were preserved. Immediate rollback is server `5848d200` with unchanged web. The separate verifier retains its real `262eae96` binary/source identity. -- [x] Accepted review queues/overlays were retired. Start created the independent language branch directly from accepted dev; #63 followed by #64, #65, #66 and #69 is the translation queue, with #68 independently included in preview checks. Completed issues are closed; five broader issues and focused #67 remain open. +- [x] Accepted review queues/overlays were retired. Start created the independent language branch directly from accepted dev; #63 followed by #64, #65, #66, #69 and #70 is the translation queue, with #68 independently included in preview checks. Completed issues are closed; five broader issues and focused #67 remain open. - [ ] Maintainer version selection, signed release commits, main promotion, tags and tag-built artifact verification. Stable releases remain v1.6.0 / v1.3.0. ## Earlier combined-candidate evidence - 20 September @@ -58,7 +58,7 @@ Language foundation #62 has since merged as accepted web dev `6d3edbb6`, source- - [x] Real preview analytics reconcile with SQL for the materialized window. Browser charts, complete-hour text, small screens and error/empty/retry states are verified. Both MQTT feeds advance and the public browser reports LIVE. - [x] Current and rollback server/web artifacts, exact source offers and visible changelog match the running pair. Only the Beacon app restarted; the other 20 containers were preserved. Additive rollup migrations retain observations and are compatible with the previous binary. -The current immediate rollback restores combined frontend `edc32842` with server `c02317a4`. Restore accepted frontend `42ba5fcb` before using the older consolidation server rollback to `5848d200`; its metadata describes the accepted frontend. The September 20 packet-reference rollback to `6be0f762` is an older recovery point. Exact artifacts/runners are retained; application rollback keeps additive rollup views and does not remove history. +The current immediate rollback restores combined frontend `8fb636aa` with server `c02317a4`. Restore accepted frontend `42ba5fcb` before using the older consolidation server rollback to `5848d200`; its metadata describes the accepted frontend. The September 20 packet-reference rollback to `6be0f762` is an older recovery point. Exact artifacts/runners are retained; application rollback keeps additive rollup views and does not remove history. ## Maintainer release handoff diff --git a/ROADMAP.md b/ROADMAP.md index 8063f8f..cf58ee0 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -25,7 +25,7 @@ Current sites: ## Accepted consolidation batch -All ten server/web contributions merged into `dev` on 24 September UTC. The accepted batch has no remaining rebase dependencies. Language foundation [web #62](https://github.com/MeshCore-Beacon/beacon-web/pull/62) has also merged as `6d3edbb6`. Signal translation [web #63](https://github.com/MeshCore-Beacon/beacon-web/pull/63), Paths translation [web #64](https://github.com/MeshCore-Beacon/beacon-web/pull/64), Traffic translation [web #65](https://github.com/MeshCore-Beacon/beacon-web/pull/65), Scopes translation [web #66](https://github.com/MeshCore-Beacon/beacon-web/pull/66), Clock Drift translation [web #69](https://github.com/MeshCore-Beacon/beacon-web/pull/69), independent Talkers fix [web #68](https://github.com/MeshCore-Beacon/beacon-web/pull/68), and [docs #5](https://github.com/MeshCore-Beacon/beacon-docs/pull/5) remain open. +All ten server/web contributions merged into `dev` on 24 September UTC. The accepted batch has no remaining rebase dependencies. Language foundation [web #62](https://github.com/MeshCore-Beacon/beacon-web/pull/62) has also merged as `6d3edbb6`. Signal translation [web #63](https://github.com/MeshCore-Beacon/beacon-web/pull/63), Paths translation [web #64](https://github.com/MeshCore-Beacon/beacon-web/pull/64), Traffic translation [web #65](https://github.com/MeshCore-Beacon/beacon-web/pull/65), Scopes translation [web #66](https://github.com/MeshCore-Beacon/beacon-web/pull/66), Clock Drift translation [web #69](https://github.com/MeshCore-Beacon/beacon-web/pull/69), shared Timestamp translation [web #70](https://github.com/MeshCore-Beacon/beacon-web/pull/70), independent Talkers fix [web #68](https://github.com/MeshCore-Beacon/beacon-web/pull/68), and [docs #5](https://github.com/MeshCore-Beacon/beacon-docs/pull/5) remain open. The original consolidation freeze is server `c02317a4ac7228d19cab498edfa1d61186c84626` / web `0f0a6ca51c7b2c3315db77954f61b30bbdeea5e2`. CI and image builds pass at those heads; web CodeQL remains skipped and is not a security scan. Current web dev has since advanced to `6d3edbb6` through #62. Stable releases are still server **v1.6.0** and web **v1.3.0**. Merged dev is not a published stable release. @@ -44,21 +44,23 @@ The original consolidation freeze is server `c02317a4ac7228d19cab498edfa1d61186c Server #156/#158 and web #54/#56/#58/#60 are closed. Server #60/#72/#99/#116 and web #12 remain open for their remaining scope. New web #67 has a complete fix in independent #68 awaiting acceptance. Archive verification establishes structure and integrity, not authenticity, SQL safety or restorability. Packet-carried ACK/TRACE/PING references are not identity or delivery guarantees. -The workflow removed the accepted entries and overlays without rebasing or force-pushing any application branch. Signal translation #63 started directly from fresh dev `6d3edbb6`; Paths translation #64 follows its exact `01c6d1aa` head. Traffic translation #65 follows #64 at `3f2a3636`. Scopes #66 follows #65 at `a928fda5`. Clock Drift #69 follows #66 at `8d81bd5d`. Merge #63, #64, #65, #66, then #69. Future overlapping follow-ups use the declared parent through the helper. Preserve actual build identities when accepted commits have identical source trees. See [the contributor workflow](CONTRIBUTOR_WORKFLOW.md) and [consolidation checklist](RELEASE-CHECKLIST.md). +The workflow removed the accepted entries and overlays without rebasing or force-pushing any application branch. Signal translation #63 started directly from fresh dev `6d3edbb6`; Paths translation #64 follows its exact `01c6d1aa` head. Traffic translation #65 follows #64 at `3f2a3636`. Scopes #66 follows #65 at `a928fda5`. Clock Drift #69 follows #66 at `8d81bd5d`. Shared Timestamp #70 follows #69 at `efd73757`. Merge #63, #64, #65, #66, #69, then #70. Future overlapping follow-ups use the declared parent through the helper. Preserve actual build identities when accepted commits have identical source trees. See [the contributor workflow](CONTRIBUTOR_WORKFLOW.md) and [consolidation checklist](RELEASE-CHECKLIST.md). ## Current issue work -**Independent correctness fix:** [#68](https://github.com/MeshCore-Beacon/beacon-web/pull/68), `2e3b5161`, remains ready directly on dev and can merge before the translation queue. It closes focused [#67](https://github.com/MeshCore-Beacon/beacon-web/issues/67) on acceptance. The independent source's 808 Windows tests and the prior combined preview's native/browser checks stay separately recorded; Talkers translation follows that same-file fix. +**Independent correctness fix:** [#68](https://github.com/MeshCore-Beacon/beacon-web/pull/68), `2e3b5161`, remains ready directly on dev and can merge before the translation queue. It closes focused [#67](https://github.com/MeshCore-Beacon/beacon-web/issues/67) on acceptance. Its 808 Windows tests and earlier combined native/browser validation stay separately recorded; Talkers translation follows that same-file fix. -[Clock Drift translation #69](https://github.com/MeshCore-Beacon/beacon-web/pull/69), `efd73757b5eae58c8e9e6b3e01f11cce4603cd85`, follows #66 `8d81bd5d`. **Translation order: #63 -> #64 -> #65 -> #66 -> #69.** It translates the caption, table headings, drift direction words and empty/error labels. Stable column IDs preserve selected sorting and focus as labels change; existing header-based callers remain compatible. Cached pending/error rows use the unavailable state, while healthy background refreshes retain valid data. +[Shared Timestamp translation #70](https://github.com/MeshCore-Beacon/beacon-web/pull/70), `f15421133e48344d2eb1ec4eee439bdb036a629f`, follows #69 `efd73757`. **Translation order: #63 -> #64 -> #65 -> #66 -> #69 -> #70.** Timestamp callers across packets, channels, nodes, observers, routes, traces and Clock Drift now use the selected language, including relative text in absolute-mode tooltips. English `5m ago` becomes French `il y a 5m`. -The PR source passes Windows build/lint and **847 tests in 96 files**, plus exact-head CI. The actual Pi source `8fb636aafaa303808e6e695f8459d3ac639f9589` also contains independent #68, and passes native build/lint and **853 tests in 97 files**. Signs, units, rounding, warning thresholds, identities/IATAs and regional query keys are preserved. Shared Timestamp formatting and relative “ago” wording remain a later slice. +The PR source passes Windows build/lint and **854 tests in 96 files**, plus exact-head CI. The actual Pi source `300ee9743254ba54f84e3b599d35c268d0281ac6` also contains independent #68, and passes native build/lint and **860 tests in 97 files**. Seven new regression cases failed first; all 51 focused checks pass. One test verifies 100 translated timestamps keep a single interval through language changes/ticks and release it on unmount. -Local/public browser checks preserve selected node order and signed magnitudes across French switching for 100 live rows, and translated controls restore worst-first sorting. Public French stays EN DIRECT. The 320px layout contains horizontal table scrolling without page overflow. Source/JS/CSS hashes match, both brokers are connected and all 23 containers are unchanged. Initial JS grows by 246 gzip bytes over the previous combined preview. Physical Safari remains untested. +Compact s/m/h/d units, flooring, future-time clamping, local absolute dates and optional milliseconds are unchanged. Direct per-page time phrases and broader date/number formatting remain separate work. Clock Drift #69's stable table column IDs remain available for later translated sortable tables. -Current Pi: server `c02317a4` / combined web `8fb636aa`. Immediate frontend rollback is `edc32842`, which retains the Talkers fix. Current/rollback artifacts and source remain; completed staging was retired. The [source/changelog](https://canadaverse.org/beacon-dev/source.html) distinguishes PR and combined-build identities. The helper retains independent #68, checks all six application PR heads and reuses validated trees without rewriting existing branches. +Local/public browser checks verify all 100 Clock Drift timestamps in English/French, preserve the exact absolute tooltip, and keep French live packets usable at 320px. Clock tables scroll within their container; neither page overflows. Source/JS/CSS hashes match, both brokers are connected and all 23 containers are unchanged. Initial JS grows by 65 gzip bytes. Physical Safari remains untested. -Next under #12: shared Timestamp wording while #68 is reviewed, then Talkers translation after its fix lands and one stack refresh. Reuse the stable column IDs from #69 for translated sortable tables. The original consolidation release freeze remains separately owned by the maintainers; history/capacity and physical Safari remain explicit gates. +Current Pi: server `c02317a4` / combined web `300ee974`. Immediate frontend rollback is `8fb636aa`, retaining the Talkers fix. Current/rollback artifacts and source remain; completed staging was retired. The [source/changelog](https://canadaverse.org/beacon-dev/source.html) distinguishes PR and combined-build identities. The helper retains independent #68, checks all seven application PR inputs, and reuses verified trees without rewriting existing branches or requesting another Pi build. Existing CodeQL remains skipped. + +Next under #12: a bounded Mesh overview translation slice, after fresh issue/review feedback. Talkers translation follows acceptance of #68 and one stack refresh. The original consolidation release freeze remains separately owned by the maintainers; history/capacity and physical Safari remain explicit gates. ## Delivered foundations @@ -82,14 +84,14 @@ The September 20 review correction moves both new aggregate APIs onto materializ September 20 validation covered native Go/PostgreSQL/HTTP behavior and **786 web tests**, plus private backup compatibility, feature-only startup failure, TLS/password files, cancellation and schema/data/sequence restoration. That review update passed 390/1280px browser checks, with ten distinct glyphs and complete-hour text; earlier chart checks also covered 320/768px. This is historical evidence for the unchanged feature code. Current revisions and corresponding-source archives are on the preview's changelog page. -The September 24 consolidation check built accepted server `c02317a4` and retained web `42ba5fcb`, identical in source to accepted `0f0a6ca5`. Its native PostgreSQL and public/browser evidence remains in the release checklist. The Pi frontend includes #63–#66 and #69 plus independent #68 as documented above; the prior combined preview is retained for rollback. The three-hour retained analytics sample does not establish 7/30-day history or production capacity. +The September 24 consolidation check built accepted server `c02317a4` and retained web `42ba5fcb`, identical in source to accepted `0f0a6ca5`. Its native PostgreSQL and public/browser evidence remains in the release checklist. The Pi frontend includes #63–#66, #69 and #70 plus independent #68 as documented above; the prior combined preview is retained for rollback. The three-hour retained analytics sample does not establish 7/30-day history or production capacity. ## Next phases The September 20 #116 investigation has a new [current-build result](https://github.com/MeshCore-Beacon/beacon-server/issues/116#issuecomment-5753626821): a 600-second unmodified Pi capture kept both feeds connected and retained 2,169 new observations, with no ping timeout, disconnect, deadline, SQLSTATE error or HTTP 5xx response. App/PostgreSQL CPU averaged 2.14%/3.96% of one core. The preceding 3h39 log likewise has no MQTT loss or deadline error. Timestamp warnings were classified separately. This did not measure callback or pool-acquisition duration and does not establish the original cause or production capacity. No application, ordering, acknowledgement or service change was made; #116 remains open. Further capture should follow a recurrence or meaningful workload change, rather than repeatedly sampling the same healthy state. 1. **Publish the coordinated consolidation releases.** Accepted-dev native/Pi/public validation and the changelog/source handoff are complete. Maintainers choose versions, create signed release commits/tags, promote main and verify release artifacts. Follow [RELEASE-CHECKLIST.md](RELEASE-CHECKLIST.md). Pause new analytics until this breakpoint; this release does not claim complete CoreScope parity. -2. **Continue listed issue #12 in small screen groups.** The English/French foundation #62 is merged; RF / Signal (#63), Paths & Hashes (#64), Traffic (#65), Scopes (#66) and Clock Drift (#69) are ready for review. Prioritize #68 for #67. Shared Timestamp wording is next while reviewed; Talkers translation follows the fix. Preserve identifiers and measurement semantics. Other page, chart and dialog text remains; keep partial coverage explicit. Refresh all open issues/reviews before choosing the next group. +2. **Continue listed issue #12 in small screen groups.** The English/French foundation #62 is merged; RF / Signal (#63), Paths & Hashes (#64), Traffic (#65), Scopes (#66), Clock Drift (#69) and shared Timestamp (#70) are ready for review. Prioritize #68 for #67. Mesh overview text is next while reviewed; Talkers translation follows the fix. Preserve identifiers and measurement semantics. Other page, chart and dialog text remains; keep partial coverage explicit. Refresh all open issues/reviews before choosing the next group. 3. **After the consolidation release: Channel Activity analytics.** Inspect existing aggregates and add bounded channel traffic/trend views. Define messages versus receptions, unknown/encrypted-channel coverage and the behavior when keys are unavailable. Keep keys and message contents out of aggregate responses; avoid raw-message paging to build statistics. 4. **Observed ambiguity and topology.** Separate static prefix conflicts from observed unresolved/ambiguous paths. Add justified route-pattern, neighbour, hop and distance views with clear provenance and bounded work. 5. **Administration and backup slices.** Non-destructive archive validation is merged in #160. Resolve browser login/session and restore authorization before an import endpoint; keep deployment-file coverage and remote/scheduled backup separate. Test restoration against disposable databases. These follow-ups do not delay the current release. @@ -104,7 +106,7 @@ The September 20 #116 investigation has a new [current-build result](https://git | [Server #60](https://github.com/MeshCore-Beacon/beacon-server/issues/60) | Remaining administration/worker/persistence behavior; account records do not establish login sessions | | [Server #72](https://github.com/MeshCore-Beacon/beacon-server/issues/72) | Download #154 and archive validation #160 are accepted; import, browser access, deployment-file coverage and remote/scheduled backup remain | | [Web #67](https://github.com/MeshCore-Beacon/beacon-web/issues/67) | Stale Talkers display states are fixed in independent #68; closes on merge into dev | -| [Web #12](https://github.com/MeshCore-Beacon/beacon-web/issues/12) | Foundation #62 is merged; Signal #63, Paths #64, Traffic #65, Scopes #66 and Clock Drift #69 are in review. Shared Timestamp wording is next; Talkers follows #68, with other screens/dialogs/formatting remaining | +| [Web #12](https://github.com/MeshCore-Beacon/beacon-web/issues/12) | Foundation #62 is merged; Signal #63, Paths #64, Traffic #65, Scopes #66, Clock Drift #69 and shared Timestamp #70 are in review. Mesh overview text is next; Talkers follows #68, with other screens/dialogs/formatting remaining | The six completed analytics/icon/map issues are closed. Refresh the six currently open issues first at every continuation; accepted partial contributions are not grounds to close broader issues. Use closing references only when a PR completes the issue's accepted scope; use related references for partial work. From 006d285dce29bc878a67debcee5b61b57ad6e25e Mon Sep 17 00:00:00 2001 From: n30nex Date: Fri, 25 Sep 2026 06:13:39 -0400 Subject: [PATCH 20/69] docs(roadmap): prioritize independent Mesh correctness fix --- RELEASE-CHECKLIST.md | 10 +++++----- ROADMAP.md | 27 ++++++++++++++------------- 2 files changed, 19 insertions(+), 18 deletions(-) diff --git a/RELEASE-CHECKLIST.md b/RELEASE-CHECKLIST.md index ff60c16..437fc27 100644 --- a/RELEASE-CHECKLIST.md +++ b/RELEASE-CHECKLIST.md @@ -26,7 +26,7 @@ The web source tree is identical to tested preview `42ba5fcb`; preserve that art - [x] Refresh #55/#57 after the accepted #52/#53 squash. Their source trees were identical after the September 20 refresh; that history-only update needs no replacement Pi artifact. - [x] All eight application PRs pass their required checks on the published heads. Native PostgreSQL tests ran. The upstream web CodeQL job remains skipped under its existing policy and is not counted as a scan. -All six server PRs and four web PRs have merged. Server #156/#158 and web #54/#56/#58/#60 are closed. Docs #5 remains open; language foundation #62 has since merged, and translations #63/#64/#65/#66/#69/#70 and independent bug fix #68 remain outside this original frozen batch. The helper retired accepted parents/overlays without rebasing or force-pushing any application branch. Future work starts from freshly fetched dev. Backend endpoints still need deployment before dependent pages on each operator's target; Pi evidence is not evidence of another deployment. +All six server PRs and four web PRs have merged. Server #156/#158 and web #54/#56/#58/#60 are closed. Docs #5 remains open; language foundation #62 has since merged, and translations #63/#64/#65/#66/#69/#70 and independent bug fixes #68/#72 remain outside this original frozen batch. The helper retired accepted parents/overlays without rebasing or force-pushing any application branch. Future work starts from freshly fetched dev. Backend endpoints still need deployment before dependent pages on each operator's target; Pi evidence is not evidence of another deployment. ## Storage and retention boundary @@ -36,7 +36,7 @@ The stated replacement direction is lz4 compression, batched deletes, per-table A consolidation release must document its actual retention behavior and capacity limits. A future production parity cutover also needs verified durable history coverage and recovery copies. Do not infer either from example configuration or the Pi's short history. -Language foundation #62 has since merged as accepted web dev `6d3edbb6`, source-equivalent to tested build `1c77f200`. The live preview combines translation #70 (following #63–#66/#69) with independent #68 at `300ee974`: native Pi build/lint and 860 tests pass, with public/browser/source checks. The Timestamp PR source passes 854 Windows tests/CI; independent #68 retains its 808-test evidence and can merge first to close #67. Immediate frontend rollback is combined `8fb636aa`. Direct per-page time phrases and broader formatting remain follow-ups; the real scope dataset is empty. The original release freeze below remains separately recorded; maintainers choose whether to advance it to include these later contributions. +Language foundation #62 has since merged as accepted web dev `6d3edbb6`, source-equivalent to tested build `1c77f200`. The live preview combines translations through #70 with independent #68/#72 at `9d96b943`: native Pi build/lint and 867 tests pass, with public/browser/source checks. Independent Mesh #72 passes 809 Windows tests/CI and Talkers #68 retains its 808-test evidence; either can merge first to close #71/#67. Immediate frontend rollback is combined `300ee974`. The real scope dataset is empty; physical Safari remains untested. The original release freeze stays separate; maintainers choose whether to advance it to these later contributions. ## Accepted-dev verification - 24 September @@ -45,7 +45,7 @@ Language foundation #62 has since merged as accepted web dev `6d3edbb6`, source- - [x] Accepted server is running on the preview. Source/asset hashes match; both feeds advance. Signal/Paths reconcile with SQL for global/regional 1/7/30-day selections, at 2-19 ms origin latency. Only three complete hours are populated; this does not prove 7/30-day history coverage. - [x] Browser Signal/Paths charts and map load with LIVE status and no captured warnings/errors. Unchanged frontend assets keep their actual `42ba5fcb` build/source identity and prior 786-test evidence; accepted `0f0a6ca5` has the identical tree. - [x] Only the Beacon app restarted; the other 22 containers and configuration/migration journal were preserved. Immediate rollback is server `5848d200` with unchanged web. The separate verifier retains its real `262eae96` binary/source identity. -- [x] Accepted review queues/overlays were retired. Start created the independent language branch directly from accepted dev; #63 followed by #64, #65, #66, #69 and #70 is the translation queue, with #68 independently included in preview checks. Completed issues are closed; five broader issues and focused #67 remain open. +- [x] Accepted review queues/overlays were retired. Start created the independent language branch directly from accepted dev; #63 followed by #64, #65, #66, #69 and #70 is the translation queue, with #68/#72 independently included in preview checks. Completed issues are closed; five broader issues and focused #67/#71 remain open. - [ ] Maintainer version selection, signed release commits, main promotion, tags and tag-built artifact verification. Stable releases remain v1.6.0 / v1.3.0. ## Earlier combined-candidate evidence - 20 September @@ -58,7 +58,7 @@ Language foundation #62 has since merged as accepted web dev `6d3edbb6`, source- - [x] Real preview analytics reconcile with SQL for the materialized window. Browser charts, complete-hour text, small screens and error/empty/retry states are verified. Both MQTT feeds advance and the public browser reports LIVE. - [x] Current and rollback server/web artifacts, exact source offers and visible changelog match the running pair. Only the Beacon app restarted; the other 20 containers were preserved. Additive rollup migrations retain observations and are compatible with the previous binary. -The current immediate rollback restores combined frontend `8fb636aa` with server `c02317a4`. Restore accepted frontend `42ba5fcb` before using the older consolidation server rollback to `5848d200`; its metadata describes the accepted frontend. The September 20 packet-reference rollback to `6be0f762` is an older recovery point. Exact artifacts/runners are retained; application rollback keeps additive rollup views and does not remove history. +The current immediate rollback restores combined frontend `300ee974` with server `c02317a4`. Restore accepted frontend `42ba5fcb` before using the older consolidation server rollback to `5848d200`; its metadata describes the accepted frontend. The September 20 packet-reference rollback to `6be0f762` is an older recovery point. Exact artifacts/runners are retained; application rollback keeps additive rollup views and does not remove history. ## Maintainer release handoff @@ -66,4 +66,4 @@ The current immediate rollback restores combined frontend `8fb636aa` with server 2. Deploy the accepted server before dependent pages and verify both endpoints on the intended deployment. The Pi is a development validation target; production cutover remains with the owner. 3. Follow the server contribution guide for a signed version/Swagger commit, dev-to-main fast-forward, tag and release CI. Web main has the prior release squash `5ac36ce` outside dev ancestry; reconcile that stable history before promotion. Do not overwrite main. 4. Verify the tag's Actions-built artifacts and matching source. Publish accurate notes, upgrade/retention guidance, known gaps and rollback instructions. Versions/tags have not been chosen by this contribution. -5. Review the five remaining issues first when resuming development: server #60/#72/#99/#116 and web #12. These broad/partial issues remain open. After the consolidation release, Channel Activity is the next analytics page; this milestone does not establish full CoreScope parity. +5. Review the seven open issues first when resuming development: server #60/#72/#99/#116 and web #12/#67/#71. Independent web #68/#72 close the two focused bugs on acceptance; the other issues remain partial. After the consolidation release, Channel Activity is the next analytics page; this milestone does not establish full CoreScope parity. diff --git a/ROADMAP.md b/ROADMAP.md index cf58ee0..f6bbc56 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -25,7 +25,7 @@ Current sites: ## Accepted consolidation batch -All ten server/web contributions merged into `dev` on 24 September UTC. The accepted batch has no remaining rebase dependencies. Language foundation [web #62](https://github.com/MeshCore-Beacon/beacon-web/pull/62) has also merged as `6d3edbb6`. Signal translation [web #63](https://github.com/MeshCore-Beacon/beacon-web/pull/63), Paths translation [web #64](https://github.com/MeshCore-Beacon/beacon-web/pull/64), Traffic translation [web #65](https://github.com/MeshCore-Beacon/beacon-web/pull/65), Scopes translation [web #66](https://github.com/MeshCore-Beacon/beacon-web/pull/66), Clock Drift translation [web #69](https://github.com/MeshCore-Beacon/beacon-web/pull/69), shared Timestamp translation [web #70](https://github.com/MeshCore-Beacon/beacon-web/pull/70), independent Talkers fix [web #68](https://github.com/MeshCore-Beacon/beacon-web/pull/68), and [docs #5](https://github.com/MeshCore-Beacon/beacon-docs/pull/5) remain open. +All ten server/web contributions merged into `dev` on 24 September UTC. The accepted batch has no remaining rebase dependencies. Language foundation [web #62](https://github.com/MeshCore-Beacon/beacon-web/pull/62) has also merged as `6d3edbb6`. Signal translation [web #63](https://github.com/MeshCore-Beacon/beacon-web/pull/63), Paths translation [web #64](https://github.com/MeshCore-Beacon/beacon-web/pull/64), Traffic translation [web #65](https://github.com/MeshCore-Beacon/beacon-web/pull/65), Scopes translation [web #66](https://github.com/MeshCore-Beacon/beacon-web/pull/66), Clock Drift translation [web #69](https://github.com/MeshCore-Beacon/beacon-web/pull/69), shared Timestamp translation [web #70](https://github.com/MeshCore-Beacon/beacon-web/pull/70), independent Talkers fix [web #68](https://github.com/MeshCore-Beacon/beacon-web/pull/68), independent Mesh fix [web #72](https://github.com/MeshCore-Beacon/beacon-web/pull/72), and [docs #5](https://github.com/MeshCore-Beacon/beacon-docs/pull/5) remain open. The original consolidation freeze is server `c02317a4ac7228d19cab498edfa1d61186c84626` / web `0f0a6ca51c7b2c3315db77954f61b30bbdeea5e2`. CI and image builds pass at those heads; web CodeQL remains skipped and is not a security scan. Current web dev has since advanced to `6d3edbb6` through #62. Stable releases are still server **v1.6.0** and web **v1.3.0**. Merged dev is not a published stable release. @@ -42,25 +42,25 @@ The original consolidation freeze is server `c02317a4ac7228d19cab498edfa1d61186c | [Web #59](https://github.com/MeshCore-Beacon/beacon-web/pull/59) | distinguish analytics navigation icons | `c2ab29a5` | | [Web #61](https://github.com/MeshCore-Beacon/beacon-web/pull/61) | omit reset and invalid node locations | `0f0a6ca5` | -Server #156/#158 and web #54/#56/#58/#60 are closed. Server #60/#72/#99/#116 and web #12 remain open for their remaining scope. New web #67 has a complete fix in independent #68 awaiting acceptance. Archive verification establishes structure and integrity, not authenticity, SQL safety or restorability. Packet-carried ACK/TRACE/PING references are not identity or delivery guarantees. +Server #156/#158 and web #54/#56/#58/#60 are closed. Server #60/#72/#99/#116 and web #12 remain open for their remaining scope. Web #67 and #71 have complete fixes in independent #68 and #72 awaiting acceptance. Archive verification establishes structure and integrity, not authenticity, SQL safety or restorability. Packet-carried ACK/TRACE/PING references are not identity or delivery guarantees. The workflow removed the accepted entries and overlays without rebasing or force-pushing any application branch. Signal translation #63 started directly from fresh dev `6d3edbb6`; Paths translation #64 follows its exact `01c6d1aa` head. Traffic translation #65 follows #64 at `3f2a3636`. Scopes #66 follows #65 at `a928fda5`. Clock Drift #69 follows #66 at `8d81bd5d`. Shared Timestamp #70 follows #69 at `efd73757`. Merge #63, #64, #65, #66, #69, then #70. Future overlapping follow-ups use the declared parent through the helper. Preserve actual build identities when accepted commits have identical source trees. See [the contributor workflow](CONTRIBUTOR_WORKFLOW.md) and [consolidation checklist](RELEASE-CHECKLIST.md). ## Current issue work -**Independent correctness fix:** [#68](https://github.com/MeshCore-Beacon/beacon-web/pull/68), `2e3b5161`, remains ready directly on dev and can merge before the translation queue. It closes focused [#67](https://github.com/MeshCore-Beacon/beacon-web/issues/67) on acceptance. Its 808 Windows tests and earlier combined native/browser validation stay separately recorded; Talkers translation follows that same-file fix. +**Correctness before Mesh translation:** [issue #71](https://github.com/MeshCore-Beacon/beacon-web/issues/71) was reproduced during the #12 audit. Mesh displayed previous-region/range data while a new query was pending; failed refreshes could leave old KPI values, 24h sparklines and payload totals visible. [PR #72](https://github.com/MeshCore-Beacon/beacon-web/pull/72), `b56632a0fc04b14c4ca4222108875484d68d6a49`, gates each panel on successful, non-placeholder data while retaining healthy same-key background refreshes and independent panels. -[Shared Timestamp translation #70](https://github.com/MeshCore-Beacon/beacon-web/pull/70), `f15421133e48344d2eb1ec4eee439bdb036a629f`, follows #69 `efd73757`. **Translation order: #63 -> #64 -> #65 -> #66 -> #69 -> #70.** Timestamp callers across packets, channels, nodes, observers, routes, traces and Clock Drift now use the selected language, including relative text in absolute-mode tooltips. English `5m ago` becomes French `il y a 5m`. +**#72 and Talkers #68 are independent on accepted dev `6d3edbb6` and can merge first, in either order.** #72 closes #71 and #68 closes #67 on acceptance; both issues remain open until then. Their own Windows evidence is 809 and 808 tests respectively. Do not append either to the translation chain, which remains **#63 -> #64 -> #65 -> #66 -> #69 -> #70**. -The PR source passes Windows build/lint and **854 tests in 96 files**, plus exact-head CI. The actual Pi source `300ee9743254ba54f84e3b599d35c268d0281ac6` also contains independent #68, and passes native build/lint and **860 tests in 97 files**. Seven new regression cases failed first; all 51 focused checks pass. One test verifies 100 translated timestamps keep a single interval through language changes/ticks and release it on unmount. +The #72 source passes Windows build/lint and **809 tests in 94 files**, plus exact-head CI. Five regressions failed first; all seven real-QueryClient tests pass, including regional/range transitions, failed overview/sparkline/payload refreshes, cache preservation, recovery, zero/empty data, background refresh and observer navigation. Query keys/windows/cache, WebSocket counters, fixed-24h KPIs/sparklines, chart calculations and population/scope semantics remain unchanged. -Compact s/m/h/d units, flooring, future-time clamping, local absolute dates and optional milliseconds are unchanged. Direct per-page time phrases and broader date/number formatting remain separate work. Clock Drift #69's stable table column IDs remain available for later translated sortable tables. +The Pi source `9d96b94325c9b8f629df1d051e545c4f50700f32` combines translations through #70 with independent fixes #68 and #72. Native build/lint and **867 tests in 98 files** pass. Controlled local browser checks verify delayed-range loading, failed cached payload refresh and recovery while healthy panels remain visible. No faults were injected into the Pi. The public preview loads all six Mesh charts and two sparklines across range changes, stays LIVE and fits at 320px with no captured production warnings/errors. Development testing emitted ECharts disposed-instance warnings during mount/unmount; the chart wrapper is unchanged. -Local/public browser checks verify all 100 Clock Drift timestamps in English/French, preserve the exact absolute tooltip, and keep French live packets usable at 320px. Clock tables scroll within their container; neither page overflows. Source/JS/CSS hashes match, both brokers are connected and all 23 containers are unchanged. Initial JS grows by 65 gzip bytes. Physical Safari remains untested. +Current Pi: server `c02317a4` / combined web `9d96b943`. Immediate frontend rollback is `300ee974`, retaining translations and the Talkers fix. Source/JS/CSS hashes and both broker connections match; the 23 containers present at deployment stayed unchanged. Completed staging was retired with current/rollback artifacts preserved. The [source/changelog](https://canadaverse.org/beacon-dev/source.html) distinguishes the independent PR and composed source. -Current Pi: server `c02317a4` / combined web `300ee974`. Immediate frontend rollback is `8fb636aa`, retaining the Talkers fix. Current/rollback artifacts and source remain; completed staging was retired. The [source/changelog](https://canadaverse.org/beacon-dev/source.html) distinguishes PR and combined-build identities. The helper retains independent #68, checks all seven application PR inputs, and reuses verified trees without rewriting existing branches or requesting another Pi build. Existing CodeQL remains skipped. +The helper keeps #68 and #72 as full independent PR/head inputs, checks all eight application inputs and reuses verified trees without branch rewrites or another Pi build. Separate manifests/states validate each independent fix. The known web CodeQL skip remains explicit. -Next under #12: a bounded Mesh overview translation slice, after fresh issue/review feedback. Talkers translation follows acceptance of #68 and one stack refresh. The original consolidation release freeze remains separately owned by the maintainers; history/capacity and physical Safari remain explicit gates. +Next: prioritize #71/#67 acceptance and closure. Resume Mesh translation after #72 lands and Talkers after #68, with one stack refresh. Observer analytics text is independent #12 work while those fixes are reviewed. The original consolidation release freeze remains with maintainers; history/capacity and physical Safari are separate gates. ## Delivered foundations @@ -84,14 +84,14 @@ The September 20 review correction moves both new aggregate APIs onto materializ September 20 validation covered native Go/PostgreSQL/HTTP behavior and **786 web tests**, plus private backup compatibility, feature-only startup failure, TLS/password files, cancellation and schema/data/sequence restoration. That review update passed 390/1280px browser checks, with ten distinct glyphs and complete-hour text; earlier chart checks also covered 320/768px. This is historical evidence for the unchanged feature code. Current revisions and corresponding-source archives are on the preview's changelog page. -The September 24 consolidation check built accepted server `c02317a4` and retained web `42ba5fcb`, identical in source to accepted `0f0a6ca5`. Its native PostgreSQL and public/browser evidence remains in the release checklist. The Pi frontend includes #63–#66, #69 and #70 plus independent #68 as documented above; the prior combined preview is retained for rollback. The three-hour retained analytics sample does not establish 7/30-day history or production capacity. +The September 24 consolidation check built accepted server `c02317a4` and retained web `42ba5fcb`, identical in source to accepted `0f0a6ca5`. Its native PostgreSQL and public/browser evidence remains in the release checklist. The Pi frontend includes #63–#66, #69 and #70 plus independent #68/#72 as documented above; the prior combined preview is retained for rollback. The three-hour retained analytics sample does not establish 7/30-day history or production capacity. ## Next phases The September 20 #116 investigation has a new [current-build result](https://github.com/MeshCore-Beacon/beacon-server/issues/116#issuecomment-5753626821): a 600-second unmodified Pi capture kept both feeds connected and retained 2,169 new observations, with no ping timeout, disconnect, deadline, SQLSTATE error or HTTP 5xx response. App/PostgreSQL CPU averaged 2.14%/3.96% of one core. The preceding 3h39 log likewise has no MQTT loss or deadline error. Timestamp warnings were classified separately. This did not measure callback or pool-acquisition duration and does not establish the original cause or production capacity. No application, ordering, acknowledgement or service change was made; #116 remains open. Further capture should follow a recurrence or meaningful workload change, rather than repeatedly sampling the same healthy state. 1. **Publish the coordinated consolidation releases.** Accepted-dev native/Pi/public validation and the changelog/source handoff are complete. Maintainers choose versions, create signed release commits/tags, promote main and verify release artifacts. Follow [RELEASE-CHECKLIST.md](RELEASE-CHECKLIST.md). Pause new analytics until this breakpoint; this release does not claim complete CoreScope parity. -2. **Continue listed issue #12 in small screen groups.** The English/French foundation #62 is merged; RF / Signal (#63), Paths & Hashes (#64), Traffic (#65), Scopes (#66), Clock Drift (#69) and shared Timestamp (#70) are ready for review. Prioritize #68 for #67. Mesh overview text is next while reviewed; Talkers translation follows the fix. Preserve identifiers and measurement semantics. Other page, chart and dialog text remains; keep partial coverage explicit. Refresh all open issues/reviews before choosing the next group. +2. **Continue listed issue #12 in small screen groups.** The English/French foundation #62 is merged; RF / Signal (#63), Paths & Hashes (#64), Traffic (#65), Scopes (#66), Clock Drift (#69) and shared Timestamp (#70) are ready for review. Prioritize independent #72/#68 for #71/#67. Mesh/Talkers translation follows each accepted fix; Observer analytics text can proceed independently. Preserve identifiers and measurement semantics. Other page, chart and dialog text remains; keep partial coverage explicit. Refresh all open issues/reviews before choosing the next group. 3. **After the consolidation release: Channel Activity analytics.** Inspect existing aggregates and add bounded channel traffic/trend views. Define messages versus receptions, unknown/encrypted-channel coverage and the behavior when keys are unavailable. Keep keys and message contents out of aggregate responses; avoid raw-message paging to build statistics. 4. **Observed ambiguity and topology.** Separate static prefix conflicts from observed unresolved/ambiguous paths. Add justified route-pattern, neighbour, hop and distance views with clear provenance and bounded work. 5. **Administration and backup slices.** Non-destructive archive validation is merged in #160. Resolve browser login/session and restore authorization before an import endpoint; keep deployment-file coverage and remote/scheduled backup separate. Test restoration against disposable databases. These follow-ups do not delay the current release. @@ -105,10 +105,11 @@ The September 20 #116 investigation has a new [current-build result](https://git | [Server #99](https://github.com/MeshCore-Beacon/beacon-server/issues/99) | Advert names and ACK/TRACE/PING references are accepted; define any remaining packet-type formats | | [Server #60](https://github.com/MeshCore-Beacon/beacon-server/issues/60) | Remaining administration/worker/persistence behavior; account records do not establish login sessions | | [Server #72](https://github.com/MeshCore-Beacon/beacon-server/issues/72) | Download #154 and archive validation #160 are accepted; import, browser access, deployment-file coverage and remote/scheduled backup remain | +| [Web #71](https://github.com/MeshCore-Beacon/beacon-web/issues/71) | Mesh stale display states are fixed in independent #72; closes on acceptance | | [Web #67](https://github.com/MeshCore-Beacon/beacon-web/issues/67) | Stale Talkers display states are fixed in independent #68; closes on merge into dev | -| [Web #12](https://github.com/MeshCore-Beacon/beacon-web/issues/12) | Foundation #62 is merged; Signal #63, Paths #64, Traffic #65, Scopes #66, Clock Drift #69 and shared Timestamp #70 are in review. Mesh overview text is next; Talkers follows #68, with other screens/dialogs/formatting remaining | +| [Web #12](https://github.com/MeshCore-Beacon/beacon-web/issues/12) | Foundation #62 is merged; Signal #63, Paths #64, Traffic #65, Scopes #66, Clock Drift #69 and shared Timestamp #70 are in review. Mesh follows #72 and Talkers follows #68; Observer analytics and other screens/dialogs/formatting remain | -The six completed analytics/icon/map issues are closed. Refresh the six currently open issues first at every continuation; accepted partial contributions are not grounds to close broader issues. Use closing references only when a PR completes the issue's accepted scope; use related references for partial work. +The six completed analytics/icon/map issues are closed. Refresh the seven currently open issues first at every continuation; accepted partial contributions are not grounds to close broader issues. Use closing references only when a PR completes the issue's accepted scope; use related references for partial work. ## Production parity matrix From 3b4fe7441bb2713a9104a9ca4bd7e53d814d82c9 Mon Sep 17 00:00:00 2001 From: n30nex Date: Sat, 26 Sep 2026 02:21:52 -0400 Subject: [PATCH 21/69] docs(roadmap): reconcile accepted application batch and upgrade gates --- RELEASE-CHECKLIST.md | 15 +++++++------- ROADMAP.md | 49 +++++++++++++++++++++----------------------- 2 files changed, 30 insertions(+), 34 deletions(-) diff --git a/RELEASE-CHECKLIST.md b/RELEASE-CHECKLIST.md index 437fc27..893f49b 100644 --- a/RELEASE-CHECKLIST.md +++ b/RELEASE-CHECKLIST.md @@ -26,17 +26,17 @@ The web source tree is identical to tested preview `42ba5fcb`; preserve that art - [x] Refresh #55/#57 after the accepted #52/#53 squash. Their source trees were identical after the September 20 refresh; that history-only update needs no replacement Pi artifact. - [x] All eight application PRs pass their required checks on the published heads. Native PostgreSQL tests ran. The upstream web CodeQL job remains skipped under its existing policy and is not counted as a scan. -All six server PRs and four web PRs have merged. Server #156/#158 and web #54/#56/#58/#60 are closed. Docs #5 remains open; language foundation #62 has since merged, and translations #63/#64/#65/#66/#69/#70 and independent bug fixes #68/#72 remain outside this original frozen batch. The helper retired accepted parents/overlays without rebasing or force-pushing any application branch. Future work starts from freshly fetched dev. Backend endpoints still need deployment before dependent pages on each operator's target; Pi evidence is not evidence of another deployment. +All original six server and four web PRs are merged. The September 25 translation batch is accepted through web #70, with #63/#64/#65/#66/#69 closed as included; #68/#72 also merged and #67/#71 are closed. New server #162/#163 and web #73 are accepted. Only docs #5 remains open. Active application manifests are empty, acceptance history remains, and no application branch was rewritten. ## Storage and retention boundary The September 17 drop-and-reset observation-partitioning design and implementation plan were explicitly superseded on September 19. They are historical reference only. Do not implement their table drop, history reset or process-local dedup replacement. -The stated replacement direction is lz4 compression, batched deletes, per-table autovacuum tuning and a seven-day default. Those changes are not present in the verified published server dev c02317a4 (its example still says 30 days). Obtain and review the replacement contribution before describing it as shipped. Coordinate append-only migration numbers with that work; the old plan's proposed 035 is not evidence that a migration exists. +Accepted server #162 now supplies batched retention deletes, per-table autovacuum tuning in migration 037 and a seven-day default packet/chat retention when unset. #163 adds migration 038, dropping per-observation endpoint snapshots and resolving against current nodes at read time. The earlier partition/reset proposal remains superseded. Compression changes were not added by these two migrations. A consolidation release must document its actual retention behavior and capacity limits. A future production parity cutover also needs verified durable history coverage and recovery copies. Do not infer either from example configuration or the Pi's short history. -Language foundation #62 has since merged as accepted web dev `6d3edbb6`, source-equivalent to tested build `1c77f200`. The live preview combines translations through #70 with independent #68/#72 at `9d96b943`: native Pi build/lint and 867 tests pass, with public/browser/source checks. Independent Mesh #72 passes 809 Windows tests/CI and Talkers #68 retains its 808-test evidence; either can merge first to close #71/#67. Immediate frontend rollback is combined `300ee974`. The real scope dataset is empty; physical Safari remains untested. The original release freeze stays separate; maintainers choose whether to advance it to these later contributions. +Current dev is server `91b4b457` / web `54b5093a`, with passing CI/image builds (server coverage/CodeQL pass; web CodeQL skipped). The Pi still runs server `c02317a4` / web `9d96b943`, equivalent to web source accepted through #72, with prior 867-test native evidence. It lacks server #162/#163 and web #73. The current frontend rollback is `300ee974`. No new deployment/migration occurred in the September 26 audit. Before upgrading, verify a database recovery checkpoint and explicit retention policy: old server queries reference the column removed by 038, so restoring only the old binary afterward is not a valid rollback. ## Accepted-dev verification - 24 September @@ -45,8 +45,7 @@ Language foundation #62 has since merged as accepted web dev `6d3edbb6`, source- - [x] Accepted server is running on the preview. Source/asset hashes match; both feeds advance. Signal/Paths reconcile with SQL for global/regional 1/7/30-day selections, at 2-19 ms origin latency. Only three complete hours are populated; this does not prove 7/30-day history coverage. - [x] Browser Signal/Paths charts and map load with LIVE status and no captured warnings/errors. Unchanged frontend assets keep their actual `42ba5fcb` build/source identity and prior 786-test evidence; accepted `0f0a6ca5` has the identical tree. - [x] Only the Beacon app restarted; the other 22 containers and configuration/migration journal were preserved. Immediate rollback is server `5848d200` with unchanged web. The separate verifier retains its real `262eae96` binary/source identity. -- [x] Accepted review queues/overlays were retired. Start created the independent language branch directly from accepted dev; #63 followed by #64, #65, #66, #69 and #70 is the translation queue, with #68/#72 independently included in preview checks. Completed issues are closed; five broader issues and focused #67/#71 remain open. -- [ ] Maintainer version selection, signed release commits, main promotion, tags and tag-built artifact verification. Stable releases remain v1.6.0 / v1.3.0. +- [x] Accepted review queues/overlays were retired, including the translation ancestors verified as included in #70. All application PRs are accepted. Only docs #5 and the five broader issues remain open. ## Earlier combined-candidate evidence - 20 September @@ -62,8 +61,8 @@ The current immediate rollback restores combined frontend `300ee974` with server ## Maintainer release handoff -1. The application review queue is accepted. Freeze server `c02317a4` / web `0f0a6ca5`, or explicitly record any newer accepted changes before release. Confirm required checks on those exact heads. -2. Deploy the accepted server before dependent pages and verify both endpoints on the intended deployment. The Pi is a development validation target; production cutover remains with the owner. +1. The application review queue is accepted. Current dev is server `91b4b457` / web `54b5093a`; choose an explicit release freeze and confirm its exact checks. The older `c02317a4` / `0f0a6ca5` proof does not validate newer schema changes. +2. Validate migration 038 and database restore/rollback on the Pi before deployment; preserve the chosen retention policy. Deploy the accepted server before dependent pages and verify both endpoints on the intended deployment. The Pi is a development validation target; production cutover remains with the owner. 3. Follow the server contribution guide for a signed version/Swagger commit, dev-to-main fast-forward, tag and release CI. Web main has the prior release squash `5ac36ce` outside dev ancestry; reconcile that stable history before promotion. Do not overwrite main. 4. Verify the tag's Actions-built artifacts and matching source. Publish accurate notes, upgrade/retention guidance, known gaps and rollback instructions. Versions/tags have not been chosen by this contribution. -5. Review the seven open issues first when resuming development: server #60/#72/#99/#116 and web #12/#67/#71. Independent web #68/#72 close the two focused bugs on acceptance; the other issues remain partial. After the consolidation release, Channel Activity is the next analytics page; this milestone does not establish full CoreScope parity. +5. Review the five open issues first when resuming development: server #60/#72/#99/#116 and web #12. Web #67/#71 are closed; broader issues remain partial. After the consolidation release, Channel Activity is the next analytics page; this milestone does not establish full CoreScope parity. diff --git a/ROADMAP.md b/ROADMAP.md index f6bbc56..e1d473d 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,6 +1,6 @@ # Beacon parity and analytics roadmap -Updated 25 September 2026 UTC. This is the working roadmap for n30nex's ongoing contributions toward CoreScope feature parity. Maintainers decide acceptance and merge order; deployment owners handle the production switch. +Updated 26 September 2026 UTC. This is the working roadmap for n30nex's ongoing contributions toward CoreScope feature parity. Maintainers decide acceptance and merge order; deployment owners handle the production switch. Refresh GitHub issues, PR feedback and branch state before starting a phase. This document is a snapshot, and linked issues/PRs are the current source of truth. @@ -25,9 +25,9 @@ Current sites: ## Accepted consolidation batch -All ten server/web contributions merged into `dev` on 24 September UTC. The accepted batch has no remaining rebase dependencies. Language foundation [web #62](https://github.com/MeshCore-Beacon/beacon-web/pull/62) has also merged as `6d3edbb6`. Signal translation [web #63](https://github.com/MeshCore-Beacon/beacon-web/pull/63), Paths translation [web #64](https://github.com/MeshCore-Beacon/beacon-web/pull/64), Traffic translation [web #65](https://github.com/MeshCore-Beacon/beacon-web/pull/65), Scopes translation [web #66](https://github.com/MeshCore-Beacon/beacon-web/pull/66), Clock Drift translation [web #69](https://github.com/MeshCore-Beacon/beacon-web/pull/69), shared Timestamp translation [web #70](https://github.com/MeshCore-Beacon/beacon-web/pull/70), independent Talkers fix [web #68](https://github.com/MeshCore-Beacon/beacon-web/pull/68), independent Mesh fix [web #72](https://github.com/MeshCore-Beacon/beacon-web/pull/72), and [docs #5](https://github.com/MeshCore-Beacon/beacon-docs/pull/5) remain open. +All ten original server/web contributions merged on September 24. The September 25 web batch is also accepted: #70 contains translations #63/#64/#65/#66/#69 plus Timestamp wording; #68 and #72 merged separately and closed #67/#71. The five earlier translation PRs were closed as included, with exact ancestry/tree equivalence verified. Web #73 and server #162/#163 then landed. **No application PR remains open.** Only [docs #5](https://github.com/MeshCore-Beacon/beacon-docs/pull/5) remains in review. -The original consolidation freeze is server `c02317a4ac7228d19cab498edfa1d61186c84626` / web `0f0a6ca51c7b2c3315db77954f61b30bbdeea5e2`. CI and image builds pass at those heads; web CodeQL remains skipped and is not a security scan. Current web dev has since advanced to `6d3edbb6` through #62. Stable releases are still server **v1.6.0** and web **v1.3.0**. Merged dev is not a published stable release. +Current accepted dev is server `91b4b457b5f233e9b90b4030e30623095e950714` / web `54b5093ac0302c7db9d51e1d7fe23570eae7cdb5`. Exact-head CI/image builds pass; server coverage/CodeQL pass and web CodeQL remains skipped. Stable releases are still server **v1.6.0** and web **v1.3.0**. The original September 24 freeze (`c02317a4` / `0f0a6ca5`) remains historical evidence, not proof that the newer server migrations are Pi-validated. | Accepted PR | Scope | Merge commit | |---|---|---| @@ -42,31 +42,34 @@ The original consolidation freeze is server `c02317a4ac7228d19cab498edfa1d61186c | [Web #59](https://github.com/MeshCore-Beacon/beacon-web/pull/59) | distinguish analytics navigation icons | `c2ab29a5` | | [Web #61](https://github.com/MeshCore-Beacon/beacon-web/pull/61) | omit reset and invalid node locations | `0f0a6ca5` | -Server #156/#158 and web #54/#56/#58/#60 are closed. Server #60/#72/#99/#116 and web #12 remain open for their remaining scope. Web #67 and #71 have complete fixes in independent #68 and #72 awaiting acceptance. Archive verification establishes structure and integrity, not authenticity, SQL safety or restorability. Packet-carried ACK/TRACE/PING references are not identity or delivery guarantees. +Server #156/#158 and web #54/#56/#58/#60 are closed. Server #60/#72/#99/#116 and web #12 remain open for their remaining scope. Web #67/#71 are closed following #68/#72. Archive verification establishes structure and integrity, not authenticity, SQL safety or restorability. Packet-carried ACK/TRACE/PING references are not identity or delivery guarantees. -The workflow removed the accepted entries and overlays without rebasing or force-pushing any application branch. Signal translation #63 started directly from fresh dev `6d3edbb6`; Paths translation #64 follows its exact `01c6d1aa` head. Traffic translation #65 follows #64 at `3f2a3636`. Scopes #66 follows #65 at `a928fda5`. Clock Drift #69 follows #66 at `8d81bd5d`. Shared Timestamp #70 follows #69 at `efd73757`. Merge #63, #64, #65, #66, #69, then #70. Future overlapping follow-ups use the declared parent through the helper. Preserve actual build identities when accepted commits have identical source trees. See [the contributor workflow](CONTRIBUTOR_WORKFLOW.md) and [consolidation checklist](RELEASE-CHECKLIST.md). +The workflow now records the full accepted translation chain via #70 and the separate #68/#72 merges. No application branches needed rebasing or force-pushing. Active entries and preview overlays are empty; acceptance history and actual deployed identities remain. Start the next feature from refreshed dev. See [the contributor workflow](CONTRIBUTOR_WORKFLOW.md) and [consolidation checklist](RELEASE-CHECKLIST.md). -## Current issue work +## September 26 repository state -**Correctness before Mesh translation:** [issue #71](https://github.com/MeshCore-Beacon/beacon-web/issues/71) was reproduced during the #12 audit. Mesh displayed previous-region/range data while a new query was pending; failed refreshes could leave old KPI values, 24h sparklines and payload totals visible. [PR #72](https://github.com/MeshCore-Beacon/beacon-web/pull/72), `b56632a0fc04b14c4ca4222108875484d68d6a49`, gates each panel on successful, non-placeholder data while retaining healthy same-key background refreshes and independent panels. - -**#72 and Talkers #68 are independent on accepted dev `6d3edbb6` and can merge first, in either order.** #72 closes #71 and #68 closes #67 on acceptance; both issues remain open until then. Their own Windows evidence is 809 and 808 tests respectively. Do not append either to the translation chain, which remains **#63 -> #64 -> #65 -> #66 -> #69 -> #70**. +| Repository | Current head | Queue / checks | +|---|---|---| +| Server | dev `91b4b457` | No open PRs; 4 open issues; CI, coverage, CodeQL and image build pass | +| Web | dev `54b5093a` | No open PRs; #12 remains; CI/image build pass, CodeQL skipped | +| Docs | main `12997ae5` | #5 remains open, ready and mergeable; no new feedback | +| Mobile | main `6336dccb` | No open PRs/issues; no new activity since May 20 or Actions runs listed | -The #72 source passes Windows build/lint and **809 tests in 94 files**, plus exact-head CI. Five regressions failed first; all seven real-QueryClient tests pass, including regional/range transitions, failed overview/sparkline/payload refreshes, cache preservation, recovery, zero/empty data, background refresh and observer navigation. Query keys/windows/cache, WebSocket counters, fixed-24h KPIs/sparklines, chart calculations and population/scope semantics remain unchanged. +Server [#162](https://github.com/MeshCore-Beacon/beacon-server/pull/162) adds batched retention, autovacuum tuning (037) and a seven-day default packet/chat retention when unset. [#163](https://github.com/MeshCore-Beacon/beacon-server/pull/163) replaces per-observation endpoint snapshots with batched read-time resolution; old packets use current node names. Migration 038 drops the snapshot column. Web [#73](https://github.com/MeshCore-Beacon/beacon-web/pull/73) folds packet summaries into the endpoint cell without duplicate advert names/empty arrows. -The Pi source `9d96b94325c9b8f629df1d051e545c4f50700f32` combines translations through #70 with independent fixes #68 and #72. Native build/lint and **867 tests in 98 files** pass. Controlled local browser checks verify delayed-range loading, failed cached payload refresh and recovery while healthy panels remain visible. No faults were injected into the Pi. The public preview loads all six Mesh charts and two sparklines across range changes, stays LIVE and fits at 320px with no captured production warnings/errors. Development testing emitted ECharts disposed-instance warnings during mount/unmount; the chart wrapper is unchanged. +Current Pi remains server `c02317a4` / web `9d96b943`. The web source equals accepted `9cd2e8f6` through #72, retaining the prior 867-test native evidence, but does not contain later #73. Server #162/#163 are also absent. API reads work and both brokers were connected at the audit. The [source/changelog](https://canadaverse.org/beacon-dev/source.html) retains actual running identities; no deployment or migration was performed during this audit. Frontend rollback remains `300ee974`. -Current Pi: server `c02317a4` / combined web `9d96b943`. Immediate frontend rollback is `300ee974`, retaining translations and the Talkers fix. Source/JS/CSS hashes and both broker connections match; the 23 containers present at deployment stayed unchanged. Completed staging was retired with current/rollback artifacts preserved. The [source/changelog](https://canadaverse.org/beacon-dev/source.html) distinguishes the independent PR and composed source. +Before upgrading, validate database recovery and the intended retention value. **The old server's queries require the column dropped by 038; binary-only rollback after that migration is incompatible.** Record a new tested release/rollback pair after native validation; do not infer that the September 24 consolidation proof covers these new changes. -The helper keeps #68 and #72 as full independent PR/head inputs, checks all eight application inputs and reuses verified trees without branch rewrites or another Pi build. Separate manifests/states validate each independent fix. The known web CodeQL skip remains explicit. +The helper's former stop on closed #63 was correct until inclusion was verified. Accepted records now cover #63/#64/#65/#66/#69 via #70 plus #68/#72 directly; active queues/overlays are empty. Fresh plans report changed dev trees need Pi builds, but did not perform them. Mesh and Talkers translation no longer have pending-PR dependencies. -Next: prioritize #71/#67 acceptance and closure. Resume Mesh translation after #72 lands and Talkers after #68, with one stack refresh. Observer analytics text is independent #12 work while those fixes are reviewed. The original consolidation release freeze remains with maintainers; history/capacity and physical Safari are separate gates. +Next: safely validate and refresh the Pi to current dev, then resume #12 with a focused Mesh/Talkers/Observer slice from fresh web dev. Maintainers still own stable release promotion and the owner owns production cutover; Channel Activity follows that breakpoint. Physical Safari/history/capacity remain separate gates. ## Delivered foundations - Faster bounded node, route, trace and clock-stat queries; list-limit validation and NULL-observation handling. Representative changes: [#111](https://github.com/MeshCore-Beacon/beacon-server/pull/111), [#118](https://github.com/MeshCore-Beacon/beacon-server/pull/118), [#120](https://github.com/MeshCore-Beacon/beacon-server/pull/120), [#122](https://github.com/MeshCore-Beacon/beacon-server/pull/122), [#124](https://github.com/MeshCore-Beacon/beacon-server/pull/124). - MQTT client isolation, proxy identity handling, API/WebSocket limits and interrupted-index recovery. Timeout attribution in #116 remains separate from these accepted fixes. -- Observer age-out, advert summaries, endpoint snapshots and companion matching, stable channel paging, packet search and shared packet links. +- Observer age-out, advert summaries, batched endpoint resolution and companion matching (snapshots superseded by #163), stable channel paging, packet search and shared packet links. - Observer activity/telemetry and comparison, regional scope statistics, runtime administration, optional foreign-repeater detection and the backup-export foundation. ## Analytics delivery and counting rules @@ -84,18 +87,14 @@ The September 20 review correction moves both new aggregate APIs onto materializ September 20 validation covered native Go/PostgreSQL/HTTP behavior and **786 web tests**, plus private backup compatibility, feature-only startup failure, TLS/password files, cancellation and schema/data/sequence restoration. That review update passed 390/1280px browser checks, with ten distinct glyphs and complete-hour text; earlier chart checks also covered 320/768px. This is historical evidence for the unchanged feature code. Current revisions and corresponding-source archives are on the preview's changelog page. -The September 24 consolidation check built accepted server `c02317a4` and retained web `42ba5fcb`, identical in source to accepted `0f0a6ca5`. Its native PostgreSQL and public/browser evidence remains in the release checklist. The Pi frontend includes #63–#66, #69 and #70 plus independent #68/#72 as documented above; the prior combined preview is retained for rollback. The three-hour retained analytics sample does not establish 7/30-day history or production capacity. +The September 24 consolidation check built accepted server `c02317a4` and retained web `42ba5fcb`, identical in source to accepted `0f0a6ca5`. Its native PostgreSQL and public/browser evidence remains in the release checklist. The Pi frontend contains the now-accepted translations and #68/#72 as documented above; the prior combined preview is retained for rollback. The three-hour retained analytics sample does not establish 7/30-day history or production capacity. ## Next phases The September 20 #116 investigation has a new [current-build result](https://github.com/MeshCore-Beacon/beacon-server/issues/116#issuecomment-5753626821): a 600-second unmodified Pi capture kept both feeds connected and retained 2,169 new observations, with no ping timeout, disconnect, deadline, SQLSTATE error or HTTP 5xx response. App/PostgreSQL CPU averaged 2.14%/3.96% of one core. The preceding 3h39 log likewise has no MQTT loss or deadline error. Timestamp warnings were classified separately. This did not measure callback or pool-acquisition duration and does not establish the original cause or production capacity. No application, ordering, acknowledgement or service change was made; #116 remains open. Further capture should follow a recurrence or meaningful workload change, rather than repeatedly sampling the same healthy state. -1. **Publish the coordinated consolidation releases.** Accepted-dev native/Pi/public validation and the changelog/source handoff are complete. Maintainers choose versions, create signed release commits/tags, promote main and verify release artifacts. Follow [RELEASE-CHECKLIST.md](RELEASE-CHECKLIST.md). Pause new analytics until this breakpoint; this release does not claim complete CoreScope parity. -2. **Continue listed issue #12 in small screen groups.** The English/French foundation #62 is merged; RF / Signal (#63), Paths & Hashes (#64), Traffic (#65), Scopes (#66), Clock Drift (#69) and shared Timestamp (#70) are ready for review. Prioritize independent #72/#68 for #71/#67. Mesh/Talkers translation follows each accepted fix; Observer analytics text can proceed independently. Preserve identifiers and measurement semantics. Other page, chart and dialog text remains; keep partial coverage explicit. Refresh all open issues/reviews before choosing the next group. -3. **After the consolidation release: Channel Activity analytics.** Inspect existing aggregates and add bounded channel traffic/trend views. Define messages versus receptions, unknown/encrypted-channel coverage and the behavior when keys are unavailable. Keep keys and message contents out of aggregate responses; avoid raw-message paging to build statistics. -4. **Observed ambiguity and topology.** Separate static prefix conflicts from observed unresolved/ambiguous paths. Add justified route-pattern, neighbour, hop and distance views with clear provenance and bounded work. -5. **Administration and backup slices.** Non-destructive archive validation is merged in #160. Resolve browser login/session and restore authorization before an import endpoint; keep deployment-file coverage and remote/scheduled backup separate. Test restoration against disposable databases. These follow-ups do not delay the current release. -6. **Production evidence and handoff.** Reconcile history, operational limits and the parity matrix below before preparing a release/cutover handoff. +1. **Validate current dev, then publish the coordinated releases.** The older consolidation pair is validated; server #162/#163 and web #73 still need Pi upgrade/recovery validation. Maintainers choose versions, create signed release commits/tags, promote main and verify release artifacts. Follow [RELEASE-CHECKLIST.md](RELEASE-CHECKLIST.md). Pause new analytics until this breakpoint; this release does not claim complete CoreScope parity. +2. **Continue listed issue #12 in small screen groups.** Foundation, Signal, Paths, Traffic, Scopes, Clock Drift and shared Timestamp work are accepted. Talkers/Mesh fixes are accepted and #67/#71 are closed. After refreshing the Pi, start Mesh, Talkers or Observer analytics translation directly from fresh dev. Preserve measurement/data semantics; broader page, chart, dialog and formatting text remains. ## Listed work still open @@ -105,11 +104,9 @@ The September 20 #116 investigation has a new [current-build result](https://git | [Server #99](https://github.com/MeshCore-Beacon/beacon-server/issues/99) | Advert names and ACK/TRACE/PING references are accepted; define any remaining packet-type formats | | [Server #60](https://github.com/MeshCore-Beacon/beacon-server/issues/60) | Remaining administration/worker/persistence behavior; account records do not establish login sessions | | [Server #72](https://github.com/MeshCore-Beacon/beacon-server/issues/72) | Download #154 and archive validation #160 are accepted; import, browser access, deployment-file coverage and remote/scheduled backup remain | -| [Web #71](https://github.com/MeshCore-Beacon/beacon-web/issues/71) | Mesh stale display states are fixed in independent #72; closes on acceptance | -| [Web #67](https://github.com/MeshCore-Beacon/beacon-web/issues/67) | Stale Talkers display states are fixed in independent #68; closes on merge into dev | -| [Web #12](https://github.com/MeshCore-Beacon/beacon-web/issues/12) | Foundation #62 is merged; Signal #63, Paths #64, Traffic #65, Scopes #66, Clock Drift #69 and shared Timestamp #70 are in review. Mesh follows #72 and Talkers follows #68; Observer analytics and other screens/dialogs/formatting remain | +| [Web #12](https://github.com/MeshCore-Beacon/beacon-web/issues/12) | Foundation and translations through #70 are accepted; Mesh/Talkers fixes are merged. Remaining analytics/screens/dialogs/formatting still need translation | -The six completed analytics/icon/map issues are closed. Refresh the seven currently open issues first at every continuation; accepted partial contributions are not grounds to close broader issues. Use closing references only when a PR completes the issue's accepted scope; use related references for partial work. +The six completed analytics/icon/map issues are closed. Refresh the five currently open issues first at every continuation; accepted partial contributions are not grounds to close broader issues. Use closing references only when a PR completes the issue's accepted scope; use related references for partial work. ## Production parity matrix From a353ba297976948e1c31e8403fc0c9eeeb474f8e Mon Sep 17 00:00:00 2001 From: n30nex Date: Sat, 26 Sep 2026 04:01:59 -0400 Subject: [PATCH 22/69] docs: record retention and endpoint preview candidates --- RELEASE-CHECKLIST.md | 20 ++++++++++++++++++++ ROADMAP.md | 27 +++++++++++++-------------- 2 files changed, 33 insertions(+), 14 deletions(-) diff --git a/RELEASE-CHECKLIST.md b/RELEASE-CHECKLIST.md index 893f49b..159876e 100644 --- a/RELEASE-CHECKLIST.md +++ b/RELEASE-CHECKLIST.md @@ -2,6 +2,26 @@ Status: accepted-dev validation and release handoff, 24 September 2026. All ten application/CLI PRs in the consolidation batch are merged. Stable tags remain unchanged; this is not a published release or a full CoreScope parity claim. +## September 26 retention and endpoint fixes + +The Pi was first matched to accepted dev server `91b4b457` / web `54b5093a`, including native validation and a verified restore across migrations 037/038. It now runs composed server `a8394f10` and web `3a18e6d1`, adding three focused review candidates: + +| PR | Head | Scope / closure | +|---|---|---| +| [Server #166](https://github.com/MeshCore-Beacon/beacon-server/pull/166) | `74f16de8` | First/renamed adverts resolve after the node update; closes #164 | +| [Server #167](https://github.com/MeshCore-Beacon/beacon-server/pull/167) | `76428b1b` | 30-day hourly summaries survive raw packet expiry; closes #165 | +| [Web #75](https://github.com/MeshCore-Beacon/beacon-web/pull/75) | `3a18e6d1` | Additional-match count and all endpoint candidates on hover, keyboard or touch; closes #74 | + +All are out of draft. Exact-head build CI passes, server CodeQL passes, and web CodeQL remains skipped. MrAlders0n/Claude review was requested in PR comments because formal review requests are unavailable to the contributor account. Both server PRs are independent on the same accepted dev and may merge in either order. Web #75 is also independent. The workflow records #167 plus #166 as a complete PR/head preview input; its separate manifest can be refreshed after upstream changes. No routine manual restacking is required for these non-overlapping changes. No upstream PR was merged by the contributor. + +The agreed Pi policy is **72-hour raw packets, 30-day hourly analytics and 720-hour telemetry**. Migration 039 archives compact summaries as each raw packet cohort expires, atomically with deletion, without storing bodies or raw paths. Traffic, payload, top observers, talkers, advertisers, observer activity, Signal and Paths use the retained summaries. Already-purged history cannot be recovered. Packet detail, sub-hour activity and exact observer comparisons still use retained raw data; entity/scope/radio population counts keep their current meaning. + +Full Windows and native Pi Go/PostgreSQL checks pass, including rollback on archive failure, retries, concurrent ingestion, late observations, distinct observers across batches, multiple IATAs, nullable/radio/path semantics and independent 30-day expiry. The full frontend build/lint and **874 tests** pass. A 1,001-packet fixture with 1KB bodies compacted to at most twelve archive rows; the first Pi run took 85ms for archive/deletion, which is a fixture measurement rather than a production-throughput guarantee. A restored copy of the actual preview database retained all eight view counts after every raw packet was deleted in a rolled-back test. Source/index hashes and the public candidate popup were verified. + +Migration 039 preserved fingerprints of all 23 original application tables. The actual prior schema038 database, exact binary/configuration and private dump remain available for rollback; older pre-038 recovery is retained separately. Only the Beacon preview app restarted (about 33 seconds); 22 other containers were unchanged and both MQTT feeds reconnected. Public admin/backup and foreign detection remain disabled. [Current changelog and corresponding source](https://canadaverse.org/beacon-dev/source.html). + +Current broader issues remain server #60 (admin), #72 (backup/import), #99 (packet summaries), #116 (MQTT investigation), and web #12 (remaining translations). Prioritize feedback and acceptance of these new fixes, then a focused Mesh/Talkers/Observer translation slice under #12. A measured month of accumulated history, production-scale capacity and physical Safari checks remain separate gates. Maintainers own stable releases and the owners handle production cutover. + ## Scope and stop point Release the accepted account/backup/analytics batch before Channel Activity or further parity expansion. Current public tags are server v1.6.0 and web v1.3.0; maintainers choose the next versions and perform signed release commits, main promotion and tags under each repository's contribution rules. diff --git a/ROADMAP.md b/ROADMAP.md index e1d473d..450c563 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -25,7 +25,7 @@ Current sites: ## Accepted consolidation batch -All ten original server/web contributions merged on September 24. The September 25 web batch is also accepted: #70 contains translations #63/#64/#65/#66/#69 plus Timestamp wording; #68 and #72 merged separately and closed #67/#71. The five earlier translation PRs were closed as included, with exact ancestry/tree equivalence verified. Web #73 and server #162/#163 then landed. **No application PR remains open.** Only [docs #5](https://github.com/MeshCore-Beacon/beacon-docs/pull/5) remains in review. +All ten original server/web contributions merged on September 24. The September 25 web batch is also accepted: #70 contains translations #63/#64/#65/#66/#69 plus Timestamp wording; #68 and #72 merged separately and closed #67/#71. The five earlier translation PRs were closed as included, with exact ancestry/tree equivalence verified. Web #73 and server #162/#163 then landed. That acceptance batch cleared the application queue. The new September 26 retention/endpoint PRs and docs #5 are now in review, as listed below. Current accepted dev is server `91b4b457b5f233e9b90b4030e30623095e950714` / web `54b5093ac0302c7db9d51e1d7fe23570eae7cdb5`. Exact-head CI/image builds pass; server coverage/CodeQL pass and web CodeQL remains skipped. Stable releases are still server **v1.6.0** and web **v1.3.0**. The original September 24 freeze (`c02317a4` / `0f0a6ca5`) remains historical evidence, not proof that the newer server migrations are Pi-validated. @@ -44,26 +44,25 @@ Current accepted dev is server `91b4b457b5f233e9b90b4030e30623095e950714` / web Server #156/#158 and web #54/#56/#58/#60 are closed. Server #60/#72/#99/#116 and web #12 remain open for their remaining scope. Web #67/#71 are closed following #68/#72. Archive verification establishes structure and integrity, not authenticity, SQL safety or restorability. Packet-carried ACK/TRACE/PING references are not identity or delivery guarantees. -The workflow now records the full accepted translation chain via #70 and the separate #68/#72 merges. No application branches needed rebasing or force-pushing. Active entries and preview overlays are empty; acceptance history and actual deployed identities remain. Start the next feature from refreshed dev. See [the contributor workflow](CONTRIBUTOR_WORKFLOW.md) and [consolidation checklist](RELEASE-CHECKLIST.md). +## September 26 retention and endpoint fixes -## September 26 repository state +The Pi was first matched to accepted dev server `91b4b457` / web `54b5093a`, including native validation and a verified restore across migrations 037/038. It now runs composed server `a8394f10` and web `3a18e6d1`, adding three focused review candidates: -| Repository | Current head | Queue / checks | +| PR | Head | Scope / closure | |---|---|---| -| Server | dev `91b4b457` | No open PRs; 4 open issues; CI, coverage, CodeQL and image build pass | -| Web | dev `54b5093a` | No open PRs; #12 remains; CI/image build pass, CodeQL skipped | -| Docs | main `12997ae5` | #5 remains open, ready and mergeable; no new feedback | -| Mobile | main `6336dccb` | No open PRs/issues; no new activity since May 20 or Actions runs listed | +| [Server #166](https://github.com/MeshCore-Beacon/beacon-server/pull/166) | `74f16de8` | First/renamed adverts resolve after the node update; closes #164 | +| [Server #167](https://github.com/MeshCore-Beacon/beacon-server/pull/167) | `76428b1b` | 30-day hourly summaries survive raw packet expiry; closes #165 | +| [Web #75](https://github.com/MeshCore-Beacon/beacon-web/pull/75) | `3a18e6d1` | Additional-match count and all endpoint candidates on hover, keyboard or touch; closes #74 | -Server [#162](https://github.com/MeshCore-Beacon/beacon-server/pull/162) adds batched retention, autovacuum tuning (037) and a seven-day default packet/chat retention when unset. [#163](https://github.com/MeshCore-Beacon/beacon-server/pull/163) replaces per-observation endpoint snapshots with batched read-time resolution; old packets use current node names. Migration 038 drops the snapshot column. Web [#73](https://github.com/MeshCore-Beacon/beacon-web/pull/73) folds packet summaries into the endpoint cell without duplicate advert names/empty arrows. +All are out of draft. Exact-head build CI passes, server CodeQL passes, and web CodeQL remains skipped. MrAlders0n/Claude review was requested in PR comments because formal review requests are unavailable to the contributor account. Both server PRs are independent on the same accepted dev and may merge in either order. Web #75 is also independent. The workflow records #167 plus #166 as a complete PR/head preview input; its separate manifest can be refreshed after upstream changes. No routine manual restacking is required for these non-overlapping changes. No upstream PR was merged by the contributor. -Current Pi remains server `c02317a4` / web `9d96b943`. The web source equals accepted `9cd2e8f6` through #72, retaining the prior 867-test native evidence, but does not contain later #73. Server #162/#163 are also absent. API reads work and both brokers were connected at the audit. The [source/changelog](https://canadaverse.org/beacon-dev/source.html) retains actual running identities; no deployment or migration was performed during this audit. Frontend rollback remains `300ee974`. +The agreed Pi policy is **72-hour raw packets, 30-day hourly analytics and 720-hour telemetry**. Migration 039 archives compact summaries as each raw packet cohort expires, atomically with deletion, without storing bodies or raw paths. Traffic, payload, top observers, talkers, advertisers, observer activity, Signal and Paths use the retained summaries. Already-purged history cannot be recovered. Packet detail, sub-hour activity and exact observer comparisons still use retained raw data; entity/scope/radio population counts keep their current meaning. -Before upgrading, validate database recovery and the intended retention value. **The old server's queries require the column dropped by 038; binary-only rollback after that migration is incompatible.** Record a new tested release/rollback pair after native validation; do not infer that the September 24 consolidation proof covers these new changes. +Full Windows and native Pi Go/PostgreSQL checks pass, including rollback on archive failure, retries, concurrent ingestion, late observations, distinct observers across batches, multiple IATAs, nullable/radio/path semantics and independent 30-day expiry. The full frontend build/lint and **874 tests** pass. A 1,001-packet fixture with 1KB bodies compacted to at most twelve archive rows; the first Pi run took 85ms for archive/deletion, which is a fixture measurement rather than a production-throughput guarantee. A restored copy of the actual preview database retained all eight view counts after every raw packet was deleted in a rolled-back test. Source/index hashes and the public candidate popup were verified. -The helper's former stop on closed #63 was correct until inclusion was verified. Accepted records now cover #63/#64/#65/#66/#69 via #70 plus #68/#72 directly; active queues/overlays are empty. Fresh plans report changed dev trees need Pi builds, but did not perform them. Mesh and Talkers translation no longer have pending-PR dependencies. +Migration 039 preserved fingerprints of all 23 original application tables. The actual prior schema038 database, exact binary/configuration and private dump remain available for rollback; older pre-038 recovery is retained separately. Only the Beacon preview app restarted (about 33 seconds); 22 other containers were unchanged and both MQTT feeds reconnected. Public admin/backup and foreign detection remain disabled. [Current changelog and corresponding source](https://canadaverse.org/beacon-dev/source.html). -Next: safely validate and refresh the Pi to current dev, then resume #12 with a focused Mesh/Talkers/Observer slice from fresh web dev. Maintainers still own stable release promotion and the owner owns production cutover; Channel Activity follows that breakpoint. Physical Safari/history/capacity remain separate gates. +Current broader issues remain server #60 (admin), #72 (backup/import), #99 (packet summaries), #116 (MQTT investigation), and web #12 (remaining translations). Prioritize feedback and acceptance of these new fixes, then a focused Mesh/Talkers/Observer translation slice under #12. A measured month of accumulated history, production-scale capacity and physical Safari checks remain separate gates. Maintainers own stable releases and the owners handle production cutover. ## Delivered foundations @@ -93,7 +92,7 @@ The September 24 consolidation check built accepted server `c02317a4` and retain The September 20 #116 investigation has a new [current-build result](https://github.com/MeshCore-Beacon/beacon-server/issues/116#issuecomment-5753626821): a 600-second unmodified Pi capture kept both feeds connected and retained 2,169 new observations, with no ping timeout, disconnect, deadline, SQLSTATE error or HTTP 5xx response. App/PostgreSQL CPU averaged 2.14%/3.96% of one core. The preceding 3h39 log likewise has no MQTT loss or deadline error. Timestamp warnings were classified separately. This did not measure callback or pool-acquisition duration and does not establish the original cause or production capacity. No application, ordering, acknowledgement or service change was made; #116 remains open. Further capture should follow a recurrence or meaningful workload change, rather than repeatedly sampling the same healthy state. -1. **Validate current dev, then publish the coordinated releases.** The older consolidation pair is validated; server #162/#163 and web #73 still need Pi upgrade/recovery validation. Maintainers choose versions, create signed release commits/tags, promote main and verify release artifacts. Follow [RELEASE-CHECKLIST.md](RELEASE-CHECKLIST.md). Pause new analytics until this breakpoint; this release does not claim complete CoreScope parity. +1. **Review the retention/endpoint fixes, then publish coordinated releases.** Accepted dev including #162/#163/#73 is now Pi-validated; #166/#167/#75 are independent review candidates on that baseline. Maintainers choose versions, create signed release commits/tags, promote main and verify release artifacts. Follow [RELEASE-CHECKLIST.md](RELEASE-CHECKLIST.md). Pause new analytics until this breakpoint; this release does not claim complete CoreScope parity. 2. **Continue listed issue #12 in small screen groups.** Foundation, Signal, Paths, Traffic, Scopes, Clock Drift and shared Timestamp work are accepted. Talkers/Mesh fixes are accepted and #67/#71 are closed. After refreshing the Pi, start Mesh, Talkers or Observer analytics translation directly from fresh dev. Preserve measurement/data semantics; broader page, chart, dialog and formatting text remains. ## Listed work still open From fa1171945b5a30d6d43baa891a2e9cabd394e140 Mon Sep 17 00:00:00 2001 From: n30nex Date: Sat, 26 Sep 2026 23:36:36 -0400 Subject: [PATCH 23/69] docs(roadmap): record observer release and public scope catalogue draft --- RELEASE-CHECKLIST.md | 34 +++++++++------ ROADMAP.md | 30 +++++++++---- app_documentation/mesh-scopes-plan.md | 19 ++++++++ app_documentation/observer-monitoring-plan.md | 43 +++++++++++++++++++ 4 files changed, 105 insertions(+), 21 deletions(-) create mode 100644 app_documentation/mesh-scopes-plan.md create mode 100644 app_documentation/observer-monitoring-plan.md diff --git a/RELEASE-CHECKLIST.md b/RELEASE-CHECKLIST.md index 159876e..95d08ef 100644 --- a/RELEASE-CHECKLIST.md +++ b/RELEASE-CHECKLIST.md @@ -1,10 +1,20 @@ # Server/web consolidation release -Status: accepted-dev validation and release handoff, 24 September 2026. All ten application/CLI PRs in the consolidation batch are merged. Stable tags remain unchanged; this is not a published release or a full CoreScope parity claim. +Status: observer-first development preview and review handoff, 27 September 2026. Stable tags and production cutover remain owner-managed. This is not a complete CoreScope parity claim. + +## Current observer release + +The observer-first release is implemented in four focused review candidates: [server #169](https://github.com/MeshCore-Beacon/beacon-server/pull/169) (`60ed339c`, closes #168), [web #79](https://github.com/MeshCore-Beacon/beacon-web/pull/79) (`3a0eb4c8`, closes #76), [web #80](https://github.com/MeshCore-Beacon/beacon-web/pull/80) (`b3a6b088`, closes #77) and [web #81](https://github.com/MeshCore-Beacon/beacon-web/pull/81) (`42ae09b7`, closes #78). All are out of draft. Server #169 follows #167; web order is #75 → #79 → #80 → #81. Independent server #166 remains in the preview composition. Maintainers control acceptance and release; these issues remain open until their changes are accepted. + +The Pi preview now runs composed server `88c2c10c830034cee70a46fca717af518803a544` and web `42ae09b7c6be50cd0f617bad8aea35825be9f12e`, with the [updated changelog and exact source](https://canadaverse.org/beacon-dev/source.html). Raw packets remain 72 hours, archived hourly analytics 30 days, telemetry 720 hours. All four candidates passed their native Pi suites; the final web build passes 895 tests. Server tests use actual PostgreSQL. Windows server/full destination/dashboard validation and 48 focused final comparison/API tests also pass. Desktop, 390-pixel phone, English/French, keyboard, legacy/shared links and Back/search/sort/scroll were checked. Physical iPhone Safari and production-volume capacity remain separate gates. + +Migration 040 preserves original rows and repairs available archived unknown-payload counts without inventing signal samples. A restored clone passed the migration probe. A fresh private dump was copied off the Pi and its checksum verified; the original schema039 database and matching binary/config/source are retained for DB-aware rollback. Only the Beacon app restarted for the server change; 22 other containers were unchanged. Frontend publication restarted no services. Both MQTT feeds reconnected. Public admin, backup and foreign detection remain disabled. + +Use the [current roadmap](ROADMAP.md) and [observer release contract](app_documentation/observer-monitoring-plan.md) for the current queue. The sections below retain earlier dated validation; their old preview/rollback identities are historical and must not be used as current deployment instructions. ## September 26 retention and endpoint fixes -The Pi was first matched to accepted dev server `91b4b457` / web `54b5093a`, including native validation and a verified restore across migrations 037/038. It now runs composed server `a8394f10` and web `3a18e6d1`, adding three focused review candidates: +The Pi was first matched to accepted dev server `91b4b457` / web `54b5093a`, including native validation and a verified restore across migrations 037/038. At that checkpoint it ran composed server `a8394f10` and web `3a18e6d1`, adding three focused review candidates: | PR | Head | Scope / closure | |---|---|---| @@ -22,7 +32,7 @@ Migration 039 preserved fingerprints of all 23 original application tables. The Current broader issues remain server #60 (admin), #72 (backup/import), #99 (packet summaries), #116 (MQTT investigation), and web #12 (remaining translations). Prioritize feedback and acceptance of these new fixes, then a focused Mesh/Talkers/Observer translation slice under #12. A measured month of accumulated history, production-scale capacity and physical Safari checks remain separate gates. Maintainers own stable releases and the owners handle production cutover. -## Scope and stop point +## Historical consolidation scope (24 September) Release the accepted account/backup/analytics batch before Channel Activity or further parity expansion. Current public tags are server v1.6.0 and web v1.3.0; maintainers choose the next versions and perform signed release commits, main promotion and tags under each repository's contribution rules. @@ -34,7 +44,7 @@ Accepted server: `c02317a4ac7228d19cab498edfa1d61186c84626`. Accepted web: `0f0a6ca51c7b2c3315db77954f61b30bbdeea5e2`. The web source tree is identical to tested preview `42ba5fcb`; preserve that artifact's actual revision/source instead of relabeling it. The server differs from preview `5848d200` only in the verifier CLI/library/tests/docs; its verifier code and tests are identical to separately tested `262eae96`. The documentation additionally contains the accepted protected-download section. -## Review gates +## Historical review gates (24 September) - [x] Workflow checks include every independent preview PR, not just the ordered stack. Status reports them; Check verifies their CI and source; Refresh/Publish reject changed prepared inputs. Server #161 and web #61 are covered by the normal preview checks. The standalone backup CLI #160 is checked with its separate manifest. - [x] Server #149: document POST/DELETE browser preflights and the full admin CORS method example. Keep public read-only defaults. @@ -46,7 +56,7 @@ The web source tree is identical to tested preview `42ba5fcb`; preserve that art - [x] Refresh #55/#57 after the accepted #52/#53 squash. Their source trees were identical after the September 20 refresh; that history-only update needs no replacement Pi artifact. - [x] All eight application PRs pass their required checks on the published heads. Native PostgreSQL tests ran. The upstream web CodeQL job remains skipped under its existing policy and is not counted as a scan. -All original six server and four web PRs are merged. The September 25 translation batch is accepted through web #70, with #63/#64/#65/#66/#69 closed as included; #68/#72 also merged and #67/#71 are closed. New server #162/#163 and web #73 are accepted. Only docs #5 remains open. Active application manifests are empty, acceptance history remains, and no application branch was rewritten. +All original six server and four web PRs are merged. The September 25 translation batch is accepted through web #70, with #63/#64/#65/#66/#69 closed as included; #68/#72 also merged and #67/#71 are closed. New server #162/#163 and web #73 are accepted. At that checkpoint only docs #5 remained open. Active application manifests are empty, acceptance history remains, and no application branch was rewritten. ## Storage and retention boundary @@ -71,18 +81,18 @@ Current dev is server `91b4b457` / web `54b5093a`, with passing CI/image builds - [x] September 20 unmodified Pi stability sample: 600 seconds / 41 samples, both feeds connected, 2,169 retained observations and no MQTT disconnect/deadline or HTTP 5xx. App/PostgreSQL CPU averaged 2.14%/3.96% of one core. This is a bounded health sample, not callback timing, #116 root-cause proof or a production-volume gate. [Result and limits](https://github.com/MeshCore-Beacon/beacon-server/issues/116#issuecomment-5753626821). - [x] Native Pi build/test of server `6be0f762` and web `42ba5fc`, including real PostgreSQL-to-HTTP checks and migration retry/concurrent refresh; 786 web tests pass. -- [x] One-million-row request/initial-population/refresh/storage measurements; request plans read only the new views. Signal: 1.8–77.5 ms reads, 14.4 s initial population, 16.8 s refresh, 6.5 MB. Paths: 3.5–141.9 ms reads, 8.4 s population, 6.2 s refresh, 11.3 MB. +- [x] One-million-row request/initial-population/refresh/storage measurements; request plans read only the new views. Signal: 1.8–77.5 ms reads, 14.4 s initial population, 16.8 s refresh, 6.5 MB. Paths: 3.5–141.9 ms reads, 8.4 s population, 6.2 s refresh, 11.3 MB. - [ ] Measure the full operator workload and sustained refresh/ingest load before a production parity claim. The million-row fixture does not establish that limit. - [x] Backup client mismatch and unsupported-DSN cases leave the public API available; valid client export/restore used disposable data only and restored all 35 source migrations. Public preview admin/backup remains disabled. - [x] Real preview analytics reconcile with SQL for the materialized window. Browser charts, complete-hour text, small screens and error/empty/retry states are verified. Both MQTT feeds advance and the public browser reports LIVE. - [x] Current and rollback server/web artifacts, exact source offers and visible changelog match the running pair. Only the Beacon app restarted; the other 20 containers were preserved. Additive rollup migrations retain observations and are compatible with the previous binary. -The current immediate rollback restores combined frontend `300ee974` with server `c02317a4`. Restore accepted frontend `42ba5fcb` before using the older consolidation server rollback to `5848d200`; its metadata describes the accepted frontend. The September 20 packet-reference rollback to `6be0f762` is an older recovery point. Exact artifacts/runners are retained; application rollback keeps additive rollup views and does not remove history. +The historical pre-retention rollback restored combined frontend `300ee974` with server `c02317a4`. Restore accepted frontend `42ba5fcb` before using the older consolidation server rollback to `5848d200`; its metadata describes the accepted frontend. The September 20 packet-reference rollback to `6be0f762` is an older recovery point. Exact artifacts/runners are retained; application rollback keeps additive rollup views and does not remove history. ## Maintainer release handoff -1. The application review queue is accepted. Current dev is server `91b4b457` / web `54b5093a`; choose an explicit release freeze and confirm its exact checks. The older `c02317a4` / `0f0a6ca5` proof does not validate newer schema changes. -2. Validate migration 038 and database restore/rollback on the Pi before deployment; preserve the chosen retention policy. Deploy the accepted server before dependent pages and verify both endpoints on the intended deployment. The Pi is a development validation target; production cutover remains with the owner. -3. Follow the server contribution guide for a signed version/Swagger commit, dev-to-main fast-forward, tag and release CI. Web main has the prior release squash `5ac36ce` outside dev ancestry; reconcile that stable history before promotion. Do not overwrite main. -4. Verify the tag's Actions-built artifacts and matching source. Publish accurate notes, upgrade/retention guidance, known gaps and rollback instructions. Versions/tags have not been chosen by this contribution. -5. Review the five open issues first when resuming development: server #60/#72/#99/#116 and web #12. Web #67/#71 are closed; broader issues remain partial. After the consolidation release, Channel Activity is the next analytics page; this milestone does not establish full CoreScope parity. +1. Review current server #166/#167/#169 and web #75/#79/#80/#81. Preserve dependencies; choose an explicit release freeze and validate its actual heads. Ready for review is not owner approval or a published release. +2. Review migration 039/040 and the verified database-aware recovery boundary. Deploy server metrics before dependent observer pages; retain packet/summary counting definitions and approved retention settings. +3. Follow each contribution guide for signed version/API commits, main promotion, tags and release CI. Reconcile stable web history instead of overwriting main. No versions or tags were chosen by this contribution. +4. Verify exact Actions-built release artifacts, corresponding source, upgrade/retention guidance and rollback on the intended deployment. Owners perform the eventual production switch. +5. Keep broader #60/#72/#99/#116 and web #12 open for their remaining scope. Continue connected investigations after feedback, with the scope-import draft separate. Physical Safari, sustained production workload and a measured month of retained history remain validation gates. diff --git a/ROADMAP.md b/ROADMAP.md index 450c563..1d266c1 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,6 +1,6 @@ # Beacon parity and analytics roadmap -Updated 26 September 2026 UTC. This is the working roadmap for n30nex's ongoing contributions toward CoreScope feature parity. Maintainers decide acceptance and merge order; deployment owners handle the production switch. +Updated 27 September 2026 UTC. This is the working roadmap for n30nex's ongoing contributions toward CoreScope feature parity. Maintainers decide acceptance and merge order; deployment owners handle the production switch. Refresh GitHub issues, PR feedback and branch state before starting a phase. This document is a snapshot, and linked issues/PRs are the current source of truth. @@ -23,6 +23,16 @@ Current sites: | [beacon-docs](https://github.com/MeshCore-Beacon/beacon-docs) | Shared contracts, operator guidance and this roadmap | `main` | | [beacon-mobile](https://github.com/MeshCore-Beacon/beacon-mobile) | Mobile client; coordinate API compatibility | `main` | +## Current observer-first release + +The observer-first release is implemented in four focused review candidates: [server #169](https://github.com/MeshCore-Beacon/beacon-server/pull/169) (`60ed339c`, closes #168), [web #79](https://github.com/MeshCore-Beacon/beacon-web/pull/79) (`3a0eb4c8`, closes #76), [web #80](https://github.com/MeshCore-Beacon/beacon-web/pull/80) (`b3a6b088`, closes #77) and [web #81](https://github.com/MeshCore-Beacon/beacon-web/pull/81) (`42ae09b7`, closes #78). All are out of draft. Server #169 follows #167; web order is #75 → #79 → #80 → #81. Independent server #166 remains in the preview composition. Maintainers control acceptance and release; these issues remain open until their changes are accepted. + +The Pi preview now runs composed server `88c2c10c830034cee70a46fca717af518803a544` and web `42ae09b7c6be50cd0f617bad8aea35825be9f12e`, with the [updated changelog and exact source](https://canadaverse.org/beacon-dev/source.html). Raw packets remain 72 hours, archived hourly analytics 30 days, telemetry 720 hours. All four candidates passed their native Pi suites; the final web build passes 895 tests. Server tests use actual PostgreSQL. Windows server/full destination/dashboard validation and 48 focused final comparison/API tests also pass. Desktop, 390-pixel phone, English/French, keyboard, legacy/shared links and Back/search/sort/scroll were checked. Physical iPhone Safari and production-volume capacity remain separate gates. + +Migration 040 preserves original rows and repairs available archived unknown-payload counts without inventing signal samples. A restored clone passed the migration probe. A fresh private dump was copied off the Pi and its checksum verified; the original schema039 database and matching binary/config/source are retained for DB-aware rollback. Only the Beacon app restarted for the server change; 22 other containers were unchanged. Frontend publication restarted no services. Both MQTT feeds reconnected. Public admin, backup and foreign detection remain disabled. + +See the [observer implementation and subsequent UX releases](app_documentation/observer-monitoring-plan.md) and the separate [Mesh Scopes interoperability draft](app_documentation/mesh-scopes-plan.md). + ## Accepted consolidation batch All ten original server/web contributions merged on September 24. The September 25 web batch is also accepted: #70 contains translations #63/#64/#65/#66/#69 plus Timestamp wording; #68 and #72 merged separately and closed #67/#71. The five earlier translation PRs were closed as included, with exact ancestry/tree equivalence verified. Web #73 and server #162/#163 then landed. That acceptance batch cleared the application queue. The new September 26 retention/endpoint PRs and docs #5 are now in review, as listed below. @@ -46,7 +56,7 @@ Server #156/#158 and web #54/#56/#58/#60 are closed. Server #60/#72/#99/#116 and ## September 26 retention and endpoint fixes -The Pi was first matched to accepted dev server `91b4b457` / web `54b5093a`, including native validation and a verified restore across migrations 037/038. It now runs composed server `a8394f10` and web `3a18e6d1`, adding three focused review candidates: +The Pi was first matched to accepted dev server `91b4b457` / web `54b5093a`, including native validation and a verified restore across migrations 037/038. At that checkpoint it ran composed server `a8394f10` and web `3a18e6d1`, adding three focused review candidates: | PR | Head | Scope / closure | |---|---|---| @@ -62,7 +72,7 @@ Full Windows and native Pi Go/PostgreSQL checks pass, including rollback on arch Migration 039 preserved fingerprints of all 23 original application tables. The actual prior schema038 database, exact binary/configuration and private dump remain available for rollback; older pre-038 recovery is retained separately. Only the Beacon preview app restarted (about 33 seconds); 22 other containers were unchanged and both MQTT feeds reconnected. Public admin/backup and foreign detection remain disabled. [Current changelog and corresponding source](https://canadaverse.org/beacon-dev/source.html). -Current broader issues remain server #60 (admin), #72 (backup/import), #99 (packet summaries), #116 (MQTT investigation), and web #12 (remaining translations). Prioritize feedback and acceptance of these new fixes, then a focused Mesh/Talkers/Observer translation slice under #12. A measured month of accumulated history, production-scale capacity and physical Safari checks remain separate gates. Maintainers own stable releases and the owners handle production cutover. +Current broader issues remain server #60 (admin), #72 (backup/import), #99 (packet summaries), #116 (MQTT investigation), and web #12 (remaining translations). Current priorities are the observer release and following investigation work described above. A measured month of accumulated history, production-scale capacity and physical Safari checks remain separate gates. Maintainers own stable releases and the owners handle production cutover. ## Delivered foundations @@ -82,7 +92,7 @@ Current broader issues remain server #60 (admin), #72 (backup/import), #99 (pack The path page counts stored receptions, not unique devices. Flood paths accumulate entries, while direct routes carry remaining entries. Observed widths do not establish device capability or collision rates. Signal readings describe reception at the reporting observer rather than a complete end-to-end link. -The September 20 review correction moves both new aggregate APIs onto materialized hourly snapshots, preserving reception/region semantics and normalizing polling windows to UTC hours. In a rolled-back million-row Pi fixture, Signal request queries took 1.8–77.5 ms and Paths 3.5–141.9 ms across custom/generic plans. Initial population took 14.4/8.4 seconds, refresh 16.8/6.2 seconds, and view/index storage was 6.5/11.3 MB respectively. These measurements cover the fixture, not the full production workload. See the [release consolidation checklist](RELEASE-CHECKLIST.md) for remaining gates. +The September 20 review correction moves both new aggregate APIs onto materialized hourly snapshots, preserving reception/region semantics and normalizing polling windows to UTC hours. In a rolled-back million-row Pi fixture, Signal request queries took 1.8–77.5 ms and Paths 3.5–141.9 ms across custom/generic plans. Initial population took 14.4/8.4 seconds, refresh 16.8/6.2 seconds, and view/index storage was 6.5/11.3 MB respectively. These measurements cover the fixture, not the full production workload. See the [release consolidation checklist](RELEASE-CHECKLIST.md) for remaining gates. September 20 validation covered native Go/PostgreSQL/HTTP behavior and **786 web tests**, plus private backup compatibility, feature-only startup failure, TLS/password files, cancellation and schema/data/sequence restoration. That review update passed 390/1280px browser checks, with ten distinct glyphs and complete-hour text; earlier chart checks also covered 320/768px. This is historical evidence for the unchanged feature code. Current revisions and corresponding-source archives are on the preview's changelog page. @@ -92,8 +102,10 @@ The September 24 consolidation check built accepted server `c02317a4` and retain The September 20 #116 investigation has a new [current-build result](https://github.com/MeshCore-Beacon/beacon-server/issues/116#issuecomment-5753626821): a 600-second unmodified Pi capture kept both feeds connected and retained 2,169 new observations, with no ping timeout, disconnect, deadline, SQLSTATE error or HTTP 5xx response. App/PostgreSQL CPU averaged 2.14%/3.96% of one core. The preceding 3h39 log likewise has no MQTT loss or deadline error. Timestamp warnings were classified separately. This did not measure callback or pool-acquisition duration and does not establish the original cause or production capacity. No application, ordering, acknowledgement or service change was made; #116 remains open. Further capture should follow a recurrence or meaningful workload change, rather than repeatedly sampling the same healthy state. -1. **Review the retention/endpoint fixes, then publish coordinated releases.** Accepted dev including #162/#163/#73 is now Pi-validated; #166/#167/#75 are independent review candidates on that baseline. Maintainers choose versions, create signed release commits/tags, promote main and verify release artifacts. Follow [RELEASE-CHECKLIST.md](RELEASE-CHECKLIST.md). Pause new analytics until this breakpoint; this release does not claim complete CoreScope parity. -2. **Continue listed issue #12 in small screen groups.** Foundation, Signal, Paths, Traffic, Scopes, Clock Drift and shared Timestamp work are accepted. Talkers/Mesh fixes are accepted and #67/#71 are closed. After refreshing the Pi, start Mesh, Talkers or Observer analytics translation directly from fresh dev. Preserve measurement/data semantics; broader page, chart, dialog and formatting text remains. +1. **Review the current retention/endpoint and observer candidates.** Keep the ordered server #167 → #169 and web #75 → #79 → #80 → #81 stacks; #166 is independent. Refresh with the existing workflow after acceptance. Maintainers choose the release breakpoint, versions, tags and main promotion. +2. **Connected investigation.** Connect packets, exact observed paths/routes, reporting observers and map actions with reliable Back navigation and visibly ambiguous identities. Continue focused issue #99/#12 work where it overlaps this accepted scope. +3. **Node/route/trace presentation, then distinct analytics questions and quality of life.** Follow the approved observer plan's subsequent releases; this phase does not claim full parity. +4. **Mesh Scopes interoperability.** Draft optional public per-IATA catalogue import while MeshMapper publishes its endpoint/schema; retain manual names and separate observed/default/imported evidence. No speculative API calls or required API key. ## Listed work still open @@ -105,7 +117,7 @@ The September 20 #116 investigation has a new [current-build result](https://git | [Server #72](https://github.com/MeshCore-Beacon/beacon-server/issues/72) | Download #154 and archive validation #160 are accepted; import, browser access, deployment-file coverage and remote/scheduled backup remain | | [Web #12](https://github.com/MeshCore-Beacon/beacon-web/issues/12) | Foundation and translations through #70 are accepted; Mesh/Talkers fixes are merged. Remaining analytics/screens/dialogs/formatting still need translation | -The six completed analytics/icon/map issues are closed. Refresh the five currently open issues first at every continuation; accepted partial contributions are not grounds to close broader issues. Use closing references only when a PR completes the issue's accepted scope; use related references for partial work. +The six completed analytics/icon/map issues are closed. Refresh all currently open issues and PR feedback first at every continuation; accepted partial contributions are not grounds to close broader issues. Use closing references only when a PR completes the issue's accepted scope; use related references for partial work. ## Production parity matrix @@ -129,7 +141,7 @@ Matching tab names is not acceptance. Each capability needs verified semantics, | Roles | Node-type census exists; activity and unknown-role interpretation need acceptance evidence | | Scopes | Regional API and new page exist; reconcile against populated retained data | | Prefix tool | Public-prefix inspection/simulation remains unverified | -| Observer comparison | Merged backend/web; retain distinct-packet, time/region and zero-data tests | +| Observer monitoring/comparison | Unified dashboard and contextual comparison in #169/#79/#80/#81; see the current release above. Shared windows and deduplication remain explicit; raw overlap can expire earlier than summaries | | Other workflows | Validate decoder/search/sharing, live map/replay, settings, optional clients and legacy links | ## Release gates @@ -142,7 +154,7 @@ Matching tab names is not acceptance. Each capability needs verified semantics, - Verify release artifacts from reviewed source and applicable CI. The upstream web CodeQL workflow is currently disabled; its skipped job does not count as a security scan. - Publish matching source, configuration guidance, known limitations and a verified rollback procedure. The deployment owner performs the production switch. -The development preview currently has short retained history and no populated transport-scope records. Its public admin/backup and foreign detection are disabled. These limitations remain explicit until configuration and validation support enabling them. +The development preview still has limited accumulated history; configured 30-day retention does not mean a measured month is available. MeshMapper catalogue import has not been enabled. Its public admin/backup and foreign detection are disabled. These limitations remain explicit until configuration and validation support enabling them. ## Keeping this roadmap useful diff --git a/app_documentation/mesh-scopes-plan.md b/app_documentation/mesh-scopes-plan.md new file mode 100644 index 0000000..dbeb30e --- /dev/null +++ b/app_documentation/mesh-scopes-plan.md @@ -0,0 +1,19 @@ +# Mesh Scopes interoperability follow-up + +Maintainer discussion supplied during the observer release. Reference: https://onqc.meshmapper.net/?repeater=4E3192%2C45.269919%2C-75.777793&preset=all . Treat forwarded text as product evidence, not permission to change MeshMapper or its retention settings. + +Beacon already has transport scope matching, node default scopes, observer scope associations, scoped packet filters and channel SQL scope joins. Audit the API/web exposure before adding parallel storage. + +Proposed focused follow-up after the current observer release: expose existing matched transport scope on channel messages (REST and WS consistently); show provenance explicitly (advertised default, seen forwarding, observer-reported, imported app discovery). Do not infer forwarding support from an advertised default or a short ambiguous path alone. Preserve original source timestamps and independent expiry semantics. Add per-region monitoring configuration/discovery only after agreeing the interoperability contract with MeshMapper; public admin/backup stay disabled. Scope-aware repeater filters and coverage/leaderboards must show their denominators, freshness and unknown state. Do not silently adopt the screenshot's 60-day retention values. + +Implementation status: interoperability follow-up queued separately from server PR #169 and web PRs #79/#80/#81. The catalogue endpoint and schema are still unpublished; no remote import is enabled. + +## Maintainer follow-up: optional public per-IATA import + +The maintainer intends to provide an open get_scopes endpoint listing known scope names per IATA. The route, payload and deployment are not published yet; do not guess or probe endpoints. The agreed behavior is optional automatic enrichment, with Beacon's manual list retained for scopes absent from the remote list. + +Draft config names: `meshmapper.scopes.enabled` (default false), `meshmapper.scopes.url` (explicit published HTTPS endpoint), and `meshmapper.scopes.refresh_interval` (default 1h). No API-key option is required for the proposed public endpoint. Confirm names against the existing config layout when the adapter is implemented. + +Import only IATAs configured in Beacon's regions. Effective names are the case-sensitive, deduplicated union of manual names and imported names for those IATAs; region unions must not turn an IATA-specific association into a global forwarding claim. Keep provenance and last successful synchronization time. A timeout, non-2xx, malformed/oversize response or unsupported schema retains the last-known-good imported catalogue and every manual scope, with an operator-visible stale state. A valid later snapshot may replace only that source's imported entries; never erase manual entries or independently observed evidence. Newly observed but unlisted scope identifiers remain explicitly unknown until named; do not invent names or overwrite manual transport keys. + +Suggested API handoff: versioned JSON with IATA and arrays of exact scope names, plus generated/updated time; public read-only access, bounded response size and conditional refresh support if available. IATA metadata sync and scope-name sync remain separate capabilities, so either can be enabled independently. Scope catalog membership is not proof a repeater forwards that scope; observed forwarding/default-scope/app-discovery evidence retains its separate timestamps and expiry. diff --git a/app_documentation/observer-monitoring-plan.md b/app_documentation/observer-monitoring-plan.md new file mode 100644 index 0000000..185a1c6 --- /dev/null +++ b/app_documentation/observer-monitoring-plan.md @@ -0,0 +1,43 @@ +# Observer monitoring release and following UX work + +Updated 27 September 2026 UTC. This records the approved observer-first plan and review implementation, not a production release or complete CoreScope parity claim. + +The observer-first release is implemented in four focused review candidates: [server #169](https://github.com/MeshCore-Beacon/beacon-server/pull/169) (`60ed339c`, closes #168), [web #79](https://github.com/MeshCore-Beacon/beacon-web/pull/79) (`3a0eb4c8`, closes #76), [web #80](https://github.com/MeshCore-Beacon/beacon-web/pull/80) (`b3a6b088`, closes #77) and [web #81](https://github.com/MeshCore-Beacon/beacon-web/pull/81) (`42ae09b7`, closes #78). All are out of draft. Server #169 follows #167; web order is #75 → #79 → #80 → #81. Independent server #166 remains in the preview composition. Maintainers control acceptance and release; these issues remain open until their changes are accepted. + +The Pi preview now runs composed server `88c2c10c830034cee70a46fca717af518803a544` and web `42ae09b7c6be50cd0f617bad8aea35825be9f12e`, with the [updated changelog and exact source](https://canadaverse.org/beacon-dev/source.html). Raw packets remain 72 hours, archived hourly analytics 30 days, telemetry 720 hours. All four candidates passed their native Pi suites; the final web build passes 895 tests. Server tests use actual PostgreSQL. Windows server/full destination/dashboard validation and 48 focused final comparison/API tests also pass. Desktop, 390-pixel phone, English/French, keyboard, legacy/shared links and Back/search/sort/scroll were checked. Physical iPhone Safari and production-volume capacity remain separate gates. + +Migration 040 preserves original rows and repairs available archived unknown-payload counts without inventing signal samples. A restored clone passed the migration probe. A fresh private dump was copied off the Pi and its checksum verified; the original schema039 database and matching binary/config/source are retained for DB-aware rollback. Only the Beacon app restarted for the server change; 22 other containers were unchanged. Frontend publication restarted no services. Both MQTT feeds reconnected. Public admin, backup and foreign detection remain disabled. + +## Four review slices + +| Slice | Delivered behavior | Dependency | +|---|---|---| +| Trustworthy metrics, server #169 | Packet-arrival bookkeeping is separate from status/neighbour presence. Activity returns its complete-bucket window, generation time, selected-window stored total, last complete hour and latest retained packet. Unknown payloads count; non-finite signal is unknown. The legacy presence counter remains API-compatible. | #167; compose independent #166 | +| Unified destination, web #79 | `?tab=Observers&observer=&range=7d`; directory selection opens the dashboard. Legacy Analytics/Stats aliases, quick inspection with Open dashboard, copied links and Back state remain compatible. | #75 | +| Dashboard, web #80 | Identity and separate freshness, six summary cards, searchable observer picker, activity/type/signal first, device telemetry below and expandable model/firmware/client/radio/brokers/key. Two-column phone cards; English/French text. | #79 and server #169 metadata | +| In-context comparison, web #81 | Current observer preselected as A; shared activity anchor/window/axes, six-metric table and retained flood-packet overlap. Invalid/self/missing/mismatched windows are explicit. Shared reader accepts legacy encoded status JSON; current noise floor and freshness remain usable. | #80 and server #169 | + +Use the existing stack refresh/publish/check helper after merges. Shared navigation/API changes stay serial; independent #166 can still merge separately. Focused parent-to-head diff links are in each PR. A history-only refresh with an identical source tree does not require a new Pi artifact. + +## Counting and discrepancy audit + +Dashboard totals count stored packet/observer records under Beacon's existing deduplication, not every RF reception or every MQTT delivery. Status/neighbour messages affect general presence and the legacy counter but never new packet-arrival timestamps. Complete bucket bounds are visible; longer ranges use coarser buckets, so their effective end can precede a shorter range's end. Summary charts may outlive raw packet details. Comparison uses the same effective window for both observers; its overlap counts retained distinct flood hashes across all received regions. Current device values/latest complete hour have their own freshness and hour bounds. + +For Orleans-Observer, the matched public key and first-seen date were verified. At the captured comparison, CoreScope's cumulative count was 358,010 versus Beacon's legacy count 356,200: **1,810 remains unexplained**. In the common completed UTC hour 2026-09-27 01:00–02:00, CoreScope showed 388 reception rows and Beacon 113 deduplicated records. These are different grains. Matching public broker names does not prove identical broker inputs, ACLs, reconnect history or persistence; the comparison is not proof of packet loss and does not close server #116. Private aligned delivery ledgers would be needed to attribute the remaining difference. No historical counter was rewritten. + +## Subsequent releases + +1. **Connected investigation:** packet → exact observed path/route → reporting observer → map. Consistent actions and return navigation; distinguish exact retained evidence from possible prefix matches. Route details need named clickable hops, related retained packets/observers, map action and shareable selection. +2. **Node, route and trace presentation:** reuse readable summaries and expandable detail. Separate current identity/location from historical activity; place a reception timeline beside traces and explain missing replies/uncertain timing without declaring loss. +3. **Additional analytics:** observer reach/timing, channel activity, route patterns and hash ambiguity, each with counting definitions and useful drill-down. Confirmed identities and unresolved prefixes must be separate; path/endpoint identifiers are not all unique nodes heard. +4. **Quality of life:** global entity search, favourites, saved packet/channel filters, explicit pause/resume/time controls and bounded retained replay. Keep map position/selection and add age/layer legends, fit-selected-path and contextual investigation. Channel key/history availability and empty-state reasons must be clear. + +Review open issues and feedback first. Broader server #60/#72/#99/#116 and web #12 remain partial. Observer translations do not close all of #12. The new [Mesh Scopes draft](mesh-scopes-plan.md) is a separate interoperability follow-up, not part of the observer PRs. + +## Acceptance retained for review + +- Six summary cards and charts reconcile to the returned effective window; recent packet traffic and stale/missing status are distinct. +- PostgreSQL regressions cover status-only messages, duplicate records, unknown payloads, archived expiry/repair and invalid samples. Existing telemetry reset/missing-data coverage remains in the full suite. +- No previous observer/range data is shown under a new selection. Invalid/old-server comparison windows show an explicit unavailable state. +- Lazy charts and shared bounded queries remain; there is no separate request per card. Observer refreshes coalesce through the query cache. +- Exact native source, public assets/source offer/changelog and rollback are recorded for the preview. Already-purged history, physical Safari, production load and owner-controlled releases remain explicit limits. From f239eb32f1bc61e807b727561c641f2c51eea54a Mon Sep 17 00:00:00 2001 From: n30nex Date: Sun, 27 Sep 2026 00:37:21 -0400 Subject: [PATCH 24/69] docs(roadmap): record packet investigation and route evidence follow-up --- RELEASE-CHECKLIST.md | 12 +++++++++++- ROADMAP.md | 12 +++++++++++- .../connected-investigation-plan.md | 19 +++++++++++++++++++ 3 files changed, 41 insertions(+), 2 deletions(-) create mode 100644 app_documentation/connected-investigation-plan.md diff --git a/RELEASE-CHECKLIST.md b/RELEASE-CHECKLIST.md index 95d08ef..4a43607 100644 --- a/RELEASE-CHECKLIST.md +++ b/RELEASE-CHECKLIST.md @@ -2,11 +2,21 @@ Status: observer-first development preview and review handoff, 27 September 2026. Stable tags and production cutover remain owner-managed. This is not a complete CoreScope parity claim. +## Packet reception investigation — 27 September + +[Web PR #83](https://github.com/MeshCore-Beacon/beacon-web/pull/83), `1d5d65e2807c2cb53a998743212d9a5b64ba80c4`, follows #81 and closes focused issue #82 when accepted. It adds grouped retained packet reports, selected-report links, observer inspection/dashboard access and a selected-path map. The initial list stays compact and keeps the selected group open. Equal prefixes are not treated as confirmed identical physical routes; empty/missing paths and TRACE intended routes have explicit labels. + +Map projection omits ambiguous/unlocated identities and breaks lines at gaps. Live animations across uncertain chains are suppressed, so fewer speculative lines appear. Unavailable selected paths no longer silently show All paths. A shared-path loading race is fixed by checking the requested packet hash. Packet labels use the existing Noto Sans stack; external basemap emoji-glyph/sprite fallback warnings can still occur. + +The Pi now runs web `1d5d65e` with unchanged server `88c2c10c`. Native build/lint and all **906 tests** pass; focused Windows checks and desktop/390px phone/English/French/keyboard/Back/shared-link checks pass. Public assets and source match, both MQTT feeds are connected, and the frontend publication restarted no services. Earlier review candidates remain included. The [changelog/source](https://canadaverse.org/beacon-dev/source.html) identifies the running build. Maintainers still own merges, stable releases and production cutover. + +**Next:** review feedback/issues first, then define a bounded known-route-to-retained-packet/report API with exact-byte versus possible-identity semantics and pagination. Non-packet overlay return navigation remains a separate follow-up. MeshMapper scope import remains a draft pending an agreed public endpoint/schema. Broader server #60/#72/#99/#116 and web #12 remain open; this is a first connected-investigation slice, not full parity. + ## Current observer release The observer-first release is implemented in four focused review candidates: [server #169](https://github.com/MeshCore-Beacon/beacon-server/pull/169) (`60ed339c`, closes #168), [web #79](https://github.com/MeshCore-Beacon/beacon-web/pull/79) (`3a0eb4c8`, closes #76), [web #80](https://github.com/MeshCore-Beacon/beacon-web/pull/80) (`b3a6b088`, closes #77) and [web #81](https://github.com/MeshCore-Beacon/beacon-web/pull/81) (`42ae09b7`, closes #78). All are out of draft. Server #169 follows #167; web order is #75 → #79 → #80 → #81. Independent server #166 remains in the preview composition. Maintainers control acceptance and release; these issues remain open until their changes are accepted. -The Pi preview now runs composed server `88c2c10c830034cee70a46fca717af518803a544` and web `42ae09b7c6be50cd0f617bad8aea35825be9f12e`, with the [updated changelog and exact source](https://canadaverse.org/beacon-dev/source.html). Raw packets remain 72 hours, archived hourly analytics 30 days, telemetry 720 hours. All four candidates passed their native Pi suites; the final web build passes 895 tests. Server tests use actual PostgreSQL. Windows server/full destination/dashboard validation and 48 focused final comparison/API tests also pass. Desktop, 390-pixel phone, English/French, keyboard, legacy/shared links and Back/search/sort/scroll were checked. Physical iPhone Safari and production-volume capacity remain separate gates. +At the observer-release checkpoint the Pi ran composed server `88c2c10c830034cee70a46fca717af518803a544` and web `42ae09b7c6be50cd0f617bad8aea35825be9f12e`, with the [updated changelog and exact source](https://canadaverse.org/beacon-dev/source.html). Raw packets remain 72 hours, archived hourly analytics 30 days, telemetry 720 hours. All four candidates passed their native Pi suites; the final web build passes 895 tests. Server tests use actual PostgreSQL. Windows server/full destination/dashboard validation and 48 focused final comparison/API tests also pass. Desktop, 390-pixel phone, English/French, keyboard, legacy/shared links and Back/search/sort/scroll were checked. Physical iPhone Safari and production-volume capacity remain separate gates. Migration 040 preserves original rows and repairs available archived unknown-payload counts without inventing signal samples. A restored clone passed the migration probe. A fresh private dump was copied off the Pi and its checksum verified; the original schema039 database and matching binary/config/source are retained for DB-aware rollback. Only the Beacon app restarted for the server change; 22 other containers were unchanged. Frontend publication restarted no services. Both MQTT feeds reconnected. Public admin, backup and foreign detection remain disabled. diff --git a/ROADMAP.md b/ROADMAP.md index 1d266c1..ca3cf9d 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -6,6 +6,16 @@ Refresh GitHub issues, PR feedback and branch state before starting a phase. Thi Release-check correction, 21 September UTC: the workflow now includes independent preview PRs in Status and Check, applies the same CI/head/fork/target requirements to them, and rechecks the prepared independent inputs before publication. This covers packet summaries #161 and map correction #61 without adding them to the ordered stacks. The backup CLI #160 retains its separate check. Twenty-three offline regressions cover these gates and the existing no-rebase/cache behavior. See [the contributor workflow](CONTRIBUTOR_WORKFLOW.md). +## Packet reception investigation — 27 September + +[Web PR #83](https://github.com/MeshCore-Beacon/beacon-web/pull/83), `1d5d65e2807c2cb53a998743212d9a5b64ba80c4`, follows #81 and closes focused issue #82 when accepted. It adds grouped retained packet reports, selected-report links, observer inspection/dashboard access and a selected-path map. The initial list stays compact and keeps the selected group open. Equal prefixes are not treated as confirmed identical physical routes; empty/missing paths and TRACE intended routes have explicit labels. + +Map projection omits ambiguous/unlocated identities and breaks lines at gaps. Live animations across uncertain chains are suppressed, so fewer speculative lines appear. Unavailable selected paths no longer silently show All paths. A shared-path loading race is fixed by checking the requested packet hash. Packet labels use the existing Noto Sans stack; external basemap emoji-glyph/sprite fallback warnings can still occur. + +The Pi now runs web `1d5d65e` with unchanged server `88c2c10c`. Native build/lint and all **906 tests** pass; focused Windows checks and desktop/390px phone/English/French/keyboard/Back/shared-link checks pass. Public assets and source match, both MQTT feeds are connected, and the frontend publication restarted no services. Earlier review candidates remain included. The [changelog/source](https://canadaverse.org/beacon-dev/source.html) identifies the running build. Maintainers still own merges, stable releases and production cutover. + +**Next:** review feedback/issues first, then define a bounded known-route-to-retained-packet/report API with exact-byte versus possible-identity semantics and pagination. Non-packet overlay return navigation remains a separate follow-up. MeshMapper scope import remains a draft pending an agreed public endpoint/schema. Broader server #60/#72/#99/#116 and web #12 remain open; this is a first connected-investigation slice, not full parity. + ## Direction Bring useful CoreScope investigation and analytics features into Beacon's existing ingest, database, API, cache and web components. Prioritize review regressions and measured stability/performance problems, then useful analytics pages. Keep each API or page a focused contribution with explicit counting semantics and validation. @@ -27,7 +37,7 @@ Current sites: The observer-first release is implemented in four focused review candidates: [server #169](https://github.com/MeshCore-Beacon/beacon-server/pull/169) (`60ed339c`, closes #168), [web #79](https://github.com/MeshCore-Beacon/beacon-web/pull/79) (`3a0eb4c8`, closes #76), [web #80](https://github.com/MeshCore-Beacon/beacon-web/pull/80) (`b3a6b088`, closes #77) and [web #81](https://github.com/MeshCore-Beacon/beacon-web/pull/81) (`42ae09b7`, closes #78). All are out of draft. Server #169 follows #167; web order is #75 → #79 → #80 → #81. Independent server #166 remains in the preview composition. Maintainers control acceptance and release; these issues remain open until their changes are accepted. -The Pi preview now runs composed server `88c2c10c830034cee70a46fca717af518803a544` and web `42ae09b7c6be50cd0f617bad8aea35825be9f12e`, with the [updated changelog and exact source](https://canadaverse.org/beacon-dev/source.html). Raw packets remain 72 hours, archived hourly analytics 30 days, telemetry 720 hours. All four candidates passed their native Pi suites; the final web build passes 895 tests. Server tests use actual PostgreSQL. Windows server/full destination/dashboard validation and 48 focused final comparison/API tests also pass. Desktop, 390-pixel phone, English/French, keyboard, legacy/shared links and Back/search/sort/scroll were checked. Physical iPhone Safari and production-volume capacity remain separate gates. +At the observer-release checkpoint the Pi ran composed server `88c2c10c830034cee70a46fca717af518803a544` and web `42ae09b7c6be50cd0f617bad8aea35825be9f12e`, with the [updated changelog and exact source](https://canadaverse.org/beacon-dev/source.html). Raw packets remain 72 hours, archived hourly analytics 30 days, telemetry 720 hours. All four candidates passed their native Pi suites; the final web build passes 895 tests. Server tests use actual PostgreSQL. Windows server/full destination/dashboard validation and 48 focused final comparison/API tests also pass. Desktop, 390-pixel phone, English/French, keyboard, legacy/shared links and Back/search/sort/scroll were checked. Physical iPhone Safari and production-volume capacity remain separate gates. Migration 040 preserves original rows and repairs available archived unknown-payload counts without inventing signal samples. A restored clone passed the migration probe. A fresh private dump was copied off the Pi and its checksum verified; the original schema039 database and matching binary/config/source are retained for DB-aware rollback. Only the Beacon app restarted for the server change; 22 other containers were unchanged. Frontend publication restarted no services. Both MQTT feeds reconnected. Public admin, backup and foreign detection remain disabled. diff --git a/app_documentation/connected-investigation-plan.md b/app_documentation/connected-investigation-plan.md new file mode 100644 index 0000000..e87878e --- /dev/null +++ b/app_documentation/connected-investigation-plan.md @@ -0,0 +1,19 @@ +# Connected investigation rollout + +## Packet reception investigation — 27 September + +[Web PR #83](https://github.com/MeshCore-Beacon/beacon-web/pull/83), `1d5d65e2807c2cb53a998743212d9a5b64ba80c4`, follows #81 and closes focused issue #82 when accepted. It adds grouped retained packet reports, selected-report links, observer inspection/dashboard access and a selected-path map. The initial list stays compact and keeps the selected group open. Equal prefixes are not treated as confirmed identical physical routes; empty/missing paths and TRACE intended routes have explicit labels. + +Map projection omits ambiguous/unlocated identities and breaks lines at gaps. Live animations across uncertain chains are suppressed, so fewer speculative lines appear. Unavailable selected paths no longer silently show All paths. A shared-path loading race is fixed by checking the requested packet hash. Packet labels use the existing Noto Sans stack; external basemap emoji-glyph/sprite fallback warnings can still occur. + +The Pi now runs web `1d5d65e` with unchanged server `88c2c10c`. Native build/lint and all **906 tests** pass; focused Windows checks and desktop/390px phone/English/French/keyboard/Back/shared-link checks pass. Public assets and source match, both MQTT feeds are connected, and the frontend publication restarted no services. Earlier review candidates remain included. The [changelog/source](https://canadaverse.org/beacon-dev/source.html) identifies the running build. Maintainers still own merges, stable releases and production cutover. + +**Next:** review feedback/issues first, then define a bounded known-route-to-retained-packet/report API with exact-byte versus possible-identity semantics and pagination. Non-packet overlay return navigation remains a separate follow-up. MeshMapper scope import remains a draft pending an agreed public endpoint/schema. Broader server #60/#72/#99/#116 and web #12 remain open; this is a first connected-investigation slice, not full parity. + +## Evidence boundaries and follow-ups + +The current packet detail is already deduplicated by packet/observer and can contain reports from multiple received regions. Grouping uses complete path bytes plus hash width; missing/malformed evidence is not folded into zero-hop paths. It is not a reconstruction of all RF receptions or an end-to-end delivery proof. The primary packet URL carries `observation=`; copied path links carry the observer path key or `trace`. Old links remain usable, and unavailable selections are explicit. + +The next backend contract should provide a bounded, paginated link from a known route to retained packet/report evidence. A match based only on endpoints or short prefixes must not be called an exact physical route. Follow that with node/route/trace presentation and richer return navigation across overlay contexts. Existing observer monitoring, retention and endpoint candidates stay in the review composition. + +Validation uses existing client/map components and no new dependency, server endpoint or migration. Physical Safari and production-volume capacity remain separate gates. The initial long report list was tightened after real browser inspection; both the preliminary and final native suites passed, but only the final source is published. From 02c7747e1116a3abc46743bcbbfaba21c889c606 Mon Sep 17 00:00:00 2001 From: n30nex Date: Sun, 27 Sep 2026 15:29:55 -0400 Subject: [PATCH 25/69] docs: record route evidence preview and next navigation issue --- CONTRIBUTOR_WORKFLOW.md | 6 ++++++ ROADMAP.md | 24 +++++++++++++++++++----- 2 files changed, 25 insertions(+), 5 deletions(-) diff --git a/CONTRIBUTOR_WORKFLOW.md b/CONTRIBUTOR_WORKFLOW.md index 1c42152..6f58837 100644 --- a/CONTRIBUTOR_WORKFLOW.md +++ b/CONTRIBUTOR_WORKFLOW.md @@ -14,6 +14,12 @@ Use this workflow to keep small Beacon changes reviewable while reducing manual A source conflict still needs review. The helper automates routine history movement; it does not promise that overlapping edits can never conflict, merge upstream PRs, deploy services, or create scheduled jobs. +## Current integration example + +The 27 September refresh moved #166 independently and #167 -> #169 together onto accepted server #170. Route evidence #172 then follows #169; its web consumer #85 follows #83. The Pi composition includes every current candidate. Refresh/Publish/Check handles this ancestry once; source conflicts still require review. A history-only change with an identical tree still needs no Pi rebuild. + +Current preview is server `99e623c5` / web `3b3abdcc`, validated natively and publicly. The route API must be deployed before its UI. Rollback retains the previous schema040 database plus previous web assets. Follow the [roadmap](ROADMAP.md) for current issues, evidence boundaries and the next navigation slice. + ## Setup Requires Python 3.10+, Git and an authenticated GitHub CLI. Server validation needs Go and Swag; web validation needs Node/npm. Use the repository's pinned dependency/toolchain requirements. diff --git a/ROADMAP.md b/ROADMAP.md index ca3cf9d..1cf1f61 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -6,15 +6,29 @@ Refresh GitHub issues, PR feedback and branch state before starting a phase. Thi Release-check correction, 21 September UTC: the workflow now includes independent preview PRs in Status and Check, applies the same CI/head/fork/target requirements to them, and rechecks the prepared independent inputs before publication. This covers packet summaries #161 and map correction #61 without adding them to the ordered stacks. The backup CLI #160 retains its separate check. Twenty-three offline regressions cover these gates and the existing no-rebase/cache behavior. See [the contributor workflow](CONTRIBUTOR_WORKFLOW.md). +## Saved-route evidence — 27 September + +[Server #172](https://github.com/MeshCore-Beacon/beacon-server/pull/172) (`f473b165`) and [web #85](https://github.com/MeshCore-Beacon/beacon-web/pull/85) (`3b3abdcc`) implement the next connected-investigation slice. They close focused server #171 / web #84 on acceptance. A full saved route now links to paged retained reports, exact packet inspection, reporting observers and named hops. Existing packet inspection leads to the selected report's map. Shared route links pin the effective server time window; browser Back and in-place inspection preserve the route/filter context. + +Matching uses the complete saved prefix bytes, width and IATA, not confirmed physical identity. Other widths, subsegments, TRACE and unclassified records are excluded. Retained reports are separate from distinct-packet or historical route totals. Empty/expired evidence and unavailable prefixes are explained. Requests default to 24 hours, permit at most 30 days and fetch 50 reports per page; the interface caps each investigation at 500. New interface text is English/French, with a compact phone layout and expandable counting definitions. + +The stack was refreshed once for accepted server #170 (`dec643a2`): optional exact WebSocket origins and opt-in observer public-key fields, not authentication. Updated server order: #167 (`02743704`) -> #169 (`91b21995`) -> #172 (`f473b165`); independent #166 (`2c5ad5fc`) remains included. Web order: #75 -> #79 -> #80 -> #81 -> #83 -> #85. All candidates remain reviewable separately; do not rebase every child independently. Deploy the server API before its web consumer. + +Current preview backend is composed `99e623c56477fd9b667d5f56bfb1a0eff34ecb2b`. Its complete native Pi/PostgreSQL suite passed. Restoring a fresh private dump and adding migration 041 preserved all 31 table fingerprints. The live switch retained the original schema040 database as `beacon_pre041_20260927`; rollback directory is `route-cutover-20260927T190644Z`. Both MQTT feeds reconnected and 22 unrelated containers were unchanged. Packets remain 72h, archived summaries 30d, telemetry 720h. Public admin, backup and foreign detection remain disabled. + +The Pi now serves web `3b3abdcc5bfa85b60c8959715736ea42c3d0bbd8`. The final native build/lint and all **915 tests** pass; exact-head CI passes (web CodeQL remains skipped). Desktop and 390px phone, English/French, copied/shared links, Back, keyboard Close/focus, exact report/node/observer and selected-map journeys were verified. All 18 public assets and both source archives match the tested artifacts. The public page is LIVE; both MQTT feeds are connected. Frontend publication restarted no containers. Web rollback retains `1d5d65e` in `web-20260927T192154Z`; the [source/changelog](https://canadaverse.org/beacon-dev/source.html) lists the complete review composition. + +**Next:** review feedback/listed issues first, then [web #86](https://github.com/MeshCore-Beacon/beacon-web/issues/86): observer quick-inspection Escape handling and broader cross-tab/overlay return navigation. The current Close button restores keyboard focus; Escape on the observer quick panel remains a reproduced follow-up. Node/trace evidence and additional reach/timing analytics follow that connection work. MeshMapper scope import remains a separate draft until the public endpoint/schema is agreed. Broad server #60/#72/#99/#116 and web #12 remain open; this phase does not claim full parity or production capacity. Maintainers retain merges, stable releases and production cutover. + ## Packet reception investigation — 27 September [Web PR #83](https://github.com/MeshCore-Beacon/beacon-web/pull/83), `1d5d65e2807c2cb53a998743212d9a5b64ba80c4`, follows #81 and closes focused issue #82 when accepted. It adds grouped retained packet reports, selected-report links, observer inspection/dashboard access and a selected-path map. The initial list stays compact and keeps the selected group open. Equal prefixes are not treated as confirmed identical physical routes; empty/missing paths and TRACE intended routes have explicit labels. Map projection omits ambiguous/unlocated identities and breaks lines at gaps. Live animations across uncertain chains are suppressed, so fewer speculative lines appear. Unavailable selected paths no longer silently show All paths. A shared-path loading race is fixed by checking the requested packet hash. Packet labels use the existing Noto Sans stack; external basemap emoji-glyph/sprite fallback warnings can still occur. -The Pi now runs web `1d5d65e` with unchanged server `88c2c10c`. Native build/lint and all **906 tests** pass; focused Windows checks and desktop/390px phone/English/French/keyboard/Back/shared-link checks pass. Public assets and source match, both MQTT feeds are connected, and the frontend publication restarted no services. Earlier review candidates remain included. The [changelog/source](https://canadaverse.org/beacon-dev/source.html) identifies the running build. Maintainers still own merges, stable releases and production cutover. +At the packet-investigation checkpoint, the Pi ran web `1d5d65e` with server `88c2c10c`. Native build/lint and all **906 tests** pass; focused Windows checks and desktop/390px phone/English/French/keyboard/Back/shared-link checks pass. Public assets and source match, both MQTT feeds are connected, and the frontend publication restarted no services. Earlier review candidates remain included. The [changelog/source](https://canadaverse.org/beacon-dev/source.html) identifies the running build. Maintainers still own merges, stable releases and production cutover. -**Next:** review feedback/issues first, then define a bounded known-route-to-retained-packet/report API with exact-byte versus possible-identity semantics and pagination. Non-packet overlay return navigation remains a separate follow-up. MeshMapper scope import remains a draft pending an agreed public endpoint/schema. Broader server #60/#72/#99/#116 and web #12 remain open; this is a first connected-investigation slice, not full parity. +**Follow-up:** the saved-route evidence phase above implements this API and interface. Remaining work is observer return navigation. Non-packet overlay return navigation remains a separate follow-up. MeshMapper scope import remains a draft pending an agreed public endpoint/schema. Broader server #60/#72/#99/#116 and web #12 remain open; this is a first connected-investigation slice, not full parity. ## Direction @@ -35,7 +49,7 @@ Current sites: ## Current observer-first release -The observer-first release is implemented in four focused review candidates: [server #169](https://github.com/MeshCore-Beacon/beacon-server/pull/169) (`60ed339c`, closes #168), [web #79](https://github.com/MeshCore-Beacon/beacon-web/pull/79) (`3a0eb4c8`, closes #76), [web #80](https://github.com/MeshCore-Beacon/beacon-web/pull/80) (`b3a6b088`, closes #77) and [web #81](https://github.com/MeshCore-Beacon/beacon-web/pull/81) (`42ae09b7`, closes #78). All are out of draft. Server #169 follows #167; web order is #75 → #79 → #80 → #81. Independent server #166 remains in the preview composition. Maintainers control acceptance and release; these issues remain open until their changes are accepted. +The observer-first release is implemented in four focused review candidates: [server #169](https://github.com/MeshCore-Beacon/beacon-server/pull/169) (`91b21995`, closes #168), [web #79](https://github.com/MeshCore-Beacon/beacon-web/pull/79) (`3a0eb4c8`, closes #76), [web #80](https://github.com/MeshCore-Beacon/beacon-web/pull/80) (`b3a6b088`, closes #77) and [web #81](https://github.com/MeshCore-Beacon/beacon-web/pull/81) (`42ae09b7`, closes #78). All are out of draft. Server #169 follows #167; web order is #75 -> #79 -> #80 -> #81. Independent server #166 remains in the preview composition. Maintainers control acceptance and release; these issues remain open until their changes are accepted. At the observer-release checkpoint the Pi ran composed server `88c2c10c830034cee70a46fca717af518803a544` and web `42ae09b7c6be50cd0f617bad8aea35825be9f12e`, with the [updated changelog and exact source](https://canadaverse.org/beacon-dev/source.html). Raw packets remain 72 hours, archived hourly analytics 30 days, telemetry 720 hours. All four candidates passed their native Pi suites; the final web build passes 895 tests. Server tests use actual PostgreSQL. Windows server/full destination/dashboard validation and 48 focused final comparison/API tests also pass. Desktop, 390-pixel phone, English/French, keyboard, legacy/shared links and Back/search/sort/scroll were checked. Physical iPhone Safari and production-volume capacity remain separate gates. @@ -47,7 +61,7 @@ See the [observer implementation and subsequent UX releases](app_documentation/o All ten original server/web contributions merged on September 24. The September 25 web batch is also accepted: #70 contains translations #63/#64/#65/#66/#69 plus Timestamp wording; #68 and #72 merged separately and closed #67/#71. The five earlier translation PRs were closed as included, with exact ancestry/tree equivalence verified. Web #73 and server #162/#163 then landed. That acceptance batch cleared the application queue. The new September 26 retention/endpoint PRs and docs #5 are now in review, as listed below. -Current accepted dev is server `91b4b457b5f233e9b90b4030e30623095e950714` / web `54b5093ac0302c7db9d51e1d7fe23570eae7cdb5`. Exact-head CI/image builds pass; server coverage/CodeQL pass and web CodeQL remains skipped. Stable releases are still server **v1.6.0** and web **v1.3.0**. The original September 24 freeze (`c02317a4` / `0f0a6ca5`) remains historical evidence, not proof that the newer server migrations are Pi-validated. +Current accepted dev is server `dec643a2ade712cd60feb2bc761a5654c7c82714` / web `54b5093ac0302c7db9d51e1d7fe23570eae7cdb5`. Exact-head CI/image builds pass; server coverage/CodeQL pass and web CodeQL remains skipped. Stable releases are still server **v1.6.0** and web **v1.3.0**. The original September 24 freeze (`c02317a4` / `0f0a6ca5`) remains historical evidence, not proof that the newer server migrations are Pi-validated. | Accepted PR | Scope | Merge commit | |---|---|---| @@ -112,7 +126,7 @@ The September 24 consolidation check built accepted server `c02317a4` and retain The September 20 #116 investigation has a new [current-build result](https://github.com/MeshCore-Beacon/beacon-server/issues/116#issuecomment-5753626821): a 600-second unmodified Pi capture kept both feeds connected and retained 2,169 new observations, with no ping timeout, disconnect, deadline, SQLSTATE error or HTTP 5xx response. App/PostgreSQL CPU averaged 2.14%/3.96% of one core. The preceding 3h39 log likewise has no MQTT loss or deadline error. Timestamp warnings were classified separately. This did not measure callback or pool-acquisition duration and does not establish the original cause or production capacity. No application, ordering, acknowledgement or service change was made; #116 remains open. Further capture should follow a recurrence or meaningful workload change, rather than repeatedly sampling the same healthy state. -1. **Review the current retention/endpoint and observer candidates.** Keep the ordered server #167 → #169 and web #75 → #79 → #80 → #81 stacks; #166 is independent. Refresh with the existing workflow after acceptance. Maintainers choose the release breakpoint, versions, tags and main promotion. +1. **Review the current retention/endpoint and observer candidates.** Keep the ordered server #167 -> #169 and web #75 -> #79 -> #80 -> #81 stacks; #166 is independent. Refresh with the existing workflow after acceptance. Maintainers choose the release breakpoint, versions, tags and main promotion. 2. **Connected investigation.** Connect packets, exact observed paths/routes, reporting observers and map actions with reliable Back navigation and visibly ambiguous identities. Continue focused issue #99/#12 work where it overlaps this accepted scope. 3. **Node/route/trace presentation, then distinct analytics questions and quality of life.** Follow the approved observer plan's subsequent releases; this phase does not claim full parity. 4. **Mesh Scopes interoperability.** Draft optional public per-IATA catalogue import while MeshMapper publishes its endpoint/schema; retain manual names and separate observed/default/imported evidence. No speculative API calls or required API key. From 7a65a6108a5e94e344f64db50fefea5f04c10af8 Mon Sep 17 00:00:00 2001 From: n30nex Date: Sun, 27 Sep 2026 17:06:43 -0400 Subject: [PATCH 26/69] docs: record observer navigation preview and next issue-first slice --- CONTRIBUTOR_WORKFLOW.md | 4 ++-- ROADMAP.md | 14 ++++++++++++-- 2 files changed, 14 insertions(+), 4 deletions(-) diff --git a/CONTRIBUTOR_WORKFLOW.md b/CONTRIBUTOR_WORKFLOW.md index 6f58837..a40609d 100644 --- a/CONTRIBUTOR_WORKFLOW.md +++ b/CONTRIBUTOR_WORKFLOW.md @@ -16,9 +16,9 @@ A source conflict still needs review. The helper automates routine history movem ## Current integration example -The 27 September refresh moved #166 independently and #167 -> #169 together onto accepted server #170. Route evidence #172 then follows #169; its web consumer #85 follows #83. The Pi composition includes every current candidate. Refresh/Publish/Check handles this ancestry once; source conflicts still require review. A history-only change with an identical tree still needs no Pi rebuild. +The 27 September refresh moved #166 independently and #167 -> #169 together onto accepted server #170. Route evidence #172 then follows #169; its web consumer #85 follows #83, and navigation #87 follows #85. The Pi composition includes every current candidate. Refresh/Publish/Check handles this ancestry once; source conflicts still require review. A history-only change with an identical tree still needs no Pi rebuild. -Current preview is server `99e623c5` / web `3b3abdcc`, validated natively and publicly. The route API must be deployed before its UI. Rollback retains the previous schema040 database plus previous web assets. Follow the [roadmap](ROADMAP.md) for current issues, evidence boundaries and the next navigation slice. +Current preview is server `99e623c5` / web `98f820d2`, validated natively (929 tests) and publicly. The route API must be deployed before its UI. Rollback retains the previous schema040 database plus previous web assets. Follow the [roadmap](ROADMAP.md) for current issues, evidence boundaries and the next navigation slice. ## Setup diff --git a/ROADMAP.md b/ROADMAP.md index 1cf1f61..c1c8cf4 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -6,6 +6,16 @@ Refresh GitHub issues, PR feedback and branch state before starting a phase. Thi Release-check correction, 21 September UTC: the workflow now includes independent preview PRs in Status and Check, applies the same CI/head/fork/target requirements to them, and rechecks the prepared independent inputs before publication. This covers packet summaries #161 and map correction #61 without adding them to the ordered stacks. The backup CLI #160 retains its separate check. Twenty-three offline regressions cover these gates and the existing no-rebase/cache behavior. See [the contributor workflow](CONTRIBUTOR_WORKFLOW.md). +## Observer investigation navigation — 27 September + +[Web #87](https://github.com/MeshCore-Beacon/beacon-web/pull/87), final candidate `98f820d2b2af5a69faf3f9aaf30d5f14b45feea8`, follows #85 and closes focused issue #86 on acceptance. Escape/Close dismisses the active panel and restores focus. Observer adverts are keyboard buttons and select the exact packet observation; embedded packet inspection preserves its originating URL. Revisiting an open entity returns to its existing panel, while tab/region/entity changes discard obsolete panels. + +Opening an observer dashboard keeps one originating screen mounted under its original Router location. Period, picker, comparison and directory detours stay within that visit; Back or the translated return action restores route/packet/node/map/Analytics state. A real route retained its typed filter, sort and 720px scroll position; a panned map's copied centre/zoom/layer/node link was identical after return. The Analytics leaderboard uses the same handler and provides keyboard buttons beside the canvas, using existing data. Direct/copied/reloaded dashboards remain standalone; arbitrary in-memory state is not persisted across reload. + +The Pi now serves web `98f820d2b2af5a69faf3f9aaf30d5f14b45feea8` on unchanged server `99e623c5`. Final native build/lint and **929 tests in 106 files** pass; exact-head CI passes (web CodeQL remains skipped). All 18 public assets and both source archives match. Public LIVE, both MQTT feeds and return/keyboard journeys pass. A real packet-list regression is fixed: the retained origin stays invisible/inert with its layout intact. Final public checks preserve 442px scroll and 506px viewport height throughout the visit, plus the selected report URL. No containers restarted. Immediate web rollback is `5a261162` at `web-20260927T210108Z`; the phase-start `3b3abdcc` recovery remains at `web-20260927T202233Z`. Existing database recovery and 72h/30d/720h retention policies are unchanged. All prior review candidates remain included. + +**Next:** review feedback and listed issues first, then the next focused web #12 translation slice for the observer directory and quick-detail panels. Node/trace presentation and further reach/timing analytics follow the approved roadmap. MeshMapper scope import still waits for an agreed public endpoint/schema. Maintainers control merges, stable releases and production cutover. + ## Saved-route evidence — 27 September [Server #172](https://github.com/MeshCore-Beacon/beacon-server/pull/172) (`f473b165`) and [web #85](https://github.com/MeshCore-Beacon/beacon-web/pull/85) (`3b3abdcc`) implement the next connected-investigation slice. They close focused server #171 / web #84 on acceptance. A full saved route now links to paged retained reports, exact packet inspection, reporting observers and named hops. Existing packet inspection leads to the selected report's map. Shared route links pin the effective server time window; browser Back and in-place inspection preserve the route/filter context. @@ -16,9 +26,9 @@ The stack was refreshed once for accepted server #170 (`dec643a2`): optional exa Current preview backend is composed `99e623c56477fd9b667d5f56bfb1a0eff34ecb2b`. Its complete native Pi/PostgreSQL suite passed. Restoring a fresh private dump and adding migration 041 preserved all 31 table fingerprints. The live switch retained the original schema040 database as `beacon_pre041_20260927`; rollback directory is `route-cutover-20260927T190644Z`. Both MQTT feeds reconnected and 22 unrelated containers were unchanged. Packets remain 72h, archived summaries 30d, telemetry 720h. Public admin, backup and foreign detection remain disabled. -The Pi now serves web `3b3abdcc5bfa85b60c8959715736ea42c3d0bbd8`. The final native build/lint and all **915 tests** pass; exact-head CI passes (web CodeQL remains skipped). Desktop and 390px phone, English/French, copied/shared links, Back, keyboard Close/focus, exact report/node/observer and selected-map journeys were verified. All 18 public assets and both source archives match the tested artifacts. The public page is LIVE; both MQTT feeds are connected. Frontend publication restarted no containers. Web rollback retains `1d5d65e` in `web-20260927T192154Z`; the [source/changelog](https://canadaverse.org/beacon-dev/source.html) lists the complete review composition. +At the route-evidence checkpoint, the Pi served web `3b3abdcc5bfa85b60c8959715736ea42c3d0bbd8`. The final native build/lint and all **915 tests** pass; exact-head CI passes (web CodeQL remains skipped). Desktop and 390px phone, English/French, copied/shared links, Back, keyboard Close/focus, exact report/node/observer and selected-map journeys were verified. All 18 public assets and both source archives match the tested artifacts. The public page is LIVE; both MQTT feeds are connected. Frontend publication restarted no containers. Web rollback retains `1d5d65e` in `web-20260927T192154Z`; the [source/changelog](https://canadaverse.org/beacon-dev/source.html) lists the complete review composition. -**Next:** review feedback/listed issues first, then [web #86](https://github.com/MeshCore-Beacon/beacon-web/issues/86): observer quick-inspection Escape handling and broader cross-tab/overlay return navigation. The current Close button restores keyboard focus; Escape on the observer quick panel remains a reproduced follow-up. Node/trace evidence and additional reach/timing analytics follow that connection work. MeshMapper scope import remains a separate draft until the public endpoint/schema is agreed. Broad server #60/#72/#99/#116 and web #12 remain open; this phase does not claim full parity or production capacity. Maintainers retain merges, stable releases and production cutover. +**Follow-up implemented above:** [web #86](https://github.com/MeshCore-Beacon/beacon-web/issues/86): observer quick-inspection Escape handling and broader cross-tab/overlay return navigation. That earlier Escape limitation is addressed by #87; acceptance remains with the maintainer. Node/trace evidence and additional reach/timing analytics follow that connection work. MeshMapper scope import remains a separate draft until the public endpoint/schema is agreed. Broad server #60/#72/#99/#116 and web #12 remain open; this phase does not claim full parity or production capacity. Maintainers retain merges, stable releases and production cutover. ## Packet reception investigation — 27 September From 89e7e08d78c008386becc308775a78ebbab69cdb Mon Sep 17 00:00:00 2001 From: n30nex Date: Sun, 27 Sep 2026 17:44:46 -0400 Subject: [PATCH 27/69] docs(scopes): record published MeshMapper API integration contract --- ROADMAP.md | 14 +++++--- app_documentation/mesh-scopes-plan.md | 47 +++++++++++++++++++++------ 2 files changed, 46 insertions(+), 15 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index c1c8cf4..013a73d 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -14,7 +14,11 @@ Opening an observer dashboard keeps one originating screen mounted under its ori The Pi now serves web `98f820d2b2af5a69faf3f9aaf30d5f14b45feea8` on unchanged server `99e623c5`. Final native build/lint and **929 tests in 106 files** pass; exact-head CI passes (web CodeQL remains skipped). All 18 public assets and both source archives match. Public LIVE, both MQTT feeds and return/keyboard journeys pass. A real packet-list regression is fixed: the retained origin stays invisible/inert with its layout intact. Final public checks preserve 442px scroll and 506px viewport height throughout the visit, plus the selected report URL. No containers restarted. Immediate web rollback is `5a261162` at `web-20260927T210108Z`; the phase-start `3b3abdcc` recovery remains at `web-20260927T202233Z`. Existing database recovery and 72h/30d/720h retention policies are unchanged. All prior review candidates remain included. -**Next:** review feedback and listed issues first, then the next focused web #12 translation slice for the observer directory and quick-detail panels. Node/trace presentation and further reach/timing analytics follow the approved roadmap. MeshMapper scope import still waits for an agreed public endpoint/schema. Maintainers control merges, stable releases and production cutover. +**Next:** review feedback and listed issues first, then the next focused web #12 translation slice for the observer directory and quick-detail panels. Node/trace presentation and further reach/timing analytics follow the approved roadmap. The public MeshMapper scopes contract is now verified; optional catalogue import is implementable as a separate follow-up, with manual scopes retained. Maintainers control merges, stable releases and production cutover. + +## MeshMapper scopes contract — 27 September + +The [public API](https://wiki.meshmapper.net/scopes-api/) is available. The documented YOW endpoint returned HTTP 200 and a successful conditional HTTP 304; it requires no API key. The [scope integration plan](app_documentation/mesh-scopes-plan.md) now specifies explicit per-IATA sources, cached refresh, durable last-known-good data, manual-name preservation and separate imported/observed evidence. Group results cannot be attributed to individual member IATAs. The former unpublished-endpoint blocker is removed; the importer and channel tags remain unimplemented. Review feedback and listed issues retain priority. This documentation update changes no application revision, Pi service or review-stack dependency. ## Saved-route evidence — 27 September @@ -28,7 +32,7 @@ Current preview backend is composed `99e623c56477fd9b667d5f56bfb1a0eff34ecb2b`. At the route-evidence checkpoint, the Pi served web `3b3abdcc5bfa85b60c8959715736ea42c3d0bbd8`. The final native build/lint and all **915 tests** pass; exact-head CI passes (web CodeQL remains skipped). Desktop and 390px phone, English/French, copied/shared links, Back, keyboard Close/focus, exact report/node/observer and selected-map journeys were verified. All 18 public assets and both source archives match the tested artifacts. The public page is LIVE; both MQTT feeds are connected. Frontend publication restarted no containers. Web rollback retains `1d5d65e` in `web-20260927T192154Z`; the [source/changelog](https://canadaverse.org/beacon-dev/source.html) lists the complete review composition. -**Follow-up implemented above:** [web #86](https://github.com/MeshCore-Beacon/beacon-web/issues/86): observer quick-inspection Escape handling and broader cross-tab/overlay return navigation. That earlier Escape limitation is addressed by #87; acceptance remains with the maintainer. Node/trace evidence and additional reach/timing analytics follow that connection work. MeshMapper scope import remains a separate draft until the public endpoint/schema is agreed. Broad server #60/#72/#99/#116 and web #12 remain open; this phase does not claim full parity or production capacity. Maintainers retain merges, stable releases and production cutover. +**Follow-up implemented above:** [web #86](https://github.com/MeshCore-Beacon/beacon-web/issues/86): observer quick-inspection Escape handling and broader cross-tab/overlay return navigation. That earlier Escape limitation is addressed by #87; acceptance remains with the maintainer. Node/trace evidence and additional reach/timing analytics follow that connection work. MeshMapper has now published its scopes contract; the separate import plan is updated, but the integration is not yet implemented. Broad server #60/#72/#99/#116 and web #12 remain open; this phase does not claim full parity or production capacity. Maintainers retain merges, stable releases and production cutover. ## Packet reception investigation — 27 September @@ -38,7 +42,7 @@ Map projection omits ambiguous/unlocated identities and breaks lines at gaps. Li At the packet-investigation checkpoint, the Pi ran web `1d5d65e` with server `88c2c10c`. Native build/lint and all **906 tests** pass; focused Windows checks and desktop/390px phone/English/French/keyboard/Back/shared-link checks pass. Public assets and source match, both MQTT feeds are connected, and the frontend publication restarted no services. Earlier review candidates remain included. The [changelog/source](https://canadaverse.org/beacon-dev/source.html) identifies the running build. Maintainers still own merges, stable releases and production cutover. -**Follow-up:** the saved-route evidence phase above implements this API and interface. Remaining work is observer return navigation. Non-packet overlay return navigation remains a separate follow-up. MeshMapper scope import remains a draft pending an agreed public endpoint/schema. Broader server #60/#72/#99/#116 and web #12 remain open; this is a first connected-investigation slice, not full parity. +**Follow-up:** the saved-route evidence phase above implements this API and interface. Remaining work is observer return navigation. Non-packet overlay return navigation remains a separate follow-up. The separate MeshMapper scope plan now uses the published and verified API; implementation remains queued. Broader server #60/#72/#99/#116 and web #12 remain open; this is a first connected-investigation slice, not full parity. ## Direction @@ -65,7 +69,7 @@ At the observer-release checkpoint the Pi ran composed server `88c2c10c830034cee Migration 040 preserves original rows and repairs available archived unknown-payload counts without inventing signal samples. A restored clone passed the migration probe. A fresh private dump was copied off the Pi and its checksum verified; the original schema039 database and matching binary/config/source are retained for DB-aware rollback. Only the Beacon app restarted for the server change; 22 other containers were unchanged. Frontend publication restarted no services. Both MQTT feeds reconnected. Public admin, backup and foreign detection remain disabled. -See the [observer implementation and subsequent UX releases](app_documentation/observer-monitoring-plan.md) and the separate [Mesh Scopes interoperability draft](app_documentation/mesh-scopes-plan.md). +See the [observer implementation and subsequent UX releases](app_documentation/observer-monitoring-plan.md) and the separate [Mesh Scopes interoperability plan](app_documentation/mesh-scopes-plan.md). ## Accepted consolidation batch @@ -139,7 +143,7 @@ The September 20 #116 investigation has a new [current-build result](https://git 1. **Review the current retention/endpoint and observer candidates.** Keep the ordered server #167 -> #169 and web #75 -> #79 -> #80 -> #81 stacks; #166 is independent. Refresh with the existing workflow after acceptance. Maintainers choose the release breakpoint, versions, tags and main promotion. 2. **Connected investigation.** Connect packets, exact observed paths/routes, reporting observers and map actions with reliable Back navigation and visibly ambiguous identities. Continue focused issue #99/#12 work where it overlaps this accepted scope. 3. **Node/route/trace presentation, then distinct analytics questions and quality of life.** Follow the approved observer plan's subsequent releases; this phase does not claim full parity. -4. **Mesh Scopes interoperability.** Draft optional public per-IATA catalogue import while MeshMapper publishes its endpoint/schema; retain manual names and separate observed/default/imported evidence. No speculative API calls or required API key. +4. **Mesh Scopes interoperability.** The published regional API is verified; implement the optional cached importer, then consistent channel scope tags, using the [integration plan](app_documentation/mesh-scopes-plan.md). Retain manual names and separate observed/default/imported evidence. No API key is required; the importer is not yet shipped. ## Listed work still open diff --git a/app_documentation/mesh-scopes-plan.md b/app_documentation/mesh-scopes-plan.md index dbeb30e..f30c587 100644 --- a/app_documentation/mesh-scopes-plan.md +++ b/app_documentation/mesh-scopes-plan.md @@ -1,19 +1,46 @@ -# Mesh Scopes interoperability follow-up +# Mesh Scopes interoperability plan -Maintainer discussion supplied during the observer release. Reference: https://onqc.meshmapper.net/?repeater=4E3192%2C45.269919%2C-75.777793&preset=all . Treat forwarded text as product evidence, not permission to change MeshMapper or its retention settings. +Updated 27 September 2026 UTC. The [MeshMapper Scopes API](https://wiki.meshmapper.net/scopes-api/) is published and its regional endpoint and conditional caching were checked live. The API-contract blocker is removed. Beacon's importer is still unimplemented and disabled; this document specifies the next focused scope work, not an available configuration feature. -Beacon already has transport scope matching, node default scopes, observer scope associations, scoped packet filters and channel SQL scope joins. Audit the API/web exposure before adding parallel storage. +## Published contract -Proposed focused follow-up after the current observer release: expose existing matched transport scope on channel messages (REST and WS consistently); show provenance explicitly (advertised default, seen forwarding, observer-reported, imported app discovery). Do not infer forwarding support from an advertised default or a short ambiguous path alone. Preserve original source timestamps and independent expiry semantics. Add per-region monitoring configuration/discovery only after agreeing the interoperability contract with MeshMapper; public admin/backup stay disabled. Scope-aware repeater filters and coverage/leaderboards must show their denominators, freshness and unknown state. Do not silently adopt the screenshot's 60-day retention values. +`GET https://yow.meshmapper.net/get_scopes.php` returns the YOW catalogue without parameters or an API key. Use an explicitly configured regional endpoint; this is not a central API accepting an IATA query parameter. Group hosts return a combined catalogue for their enabled member regions. -Implementation status: interoperability follow-up queued separately from server PR #169 and web PRs #79/#80/#81. The catalogue endpoint and schema are still unpublished; no remote import is enabled. +The response carries `generated_at`, `region`, `zones`, regional `repeaters` and `scoped` totals, and a `scopes` array. Each scope has its exact `name`, `repeaters`, `default`, `monitored` and `wardriving` values. Zero-repeater monitored or wardriving entries are useful discovery candidates and must be retained. Do not sum per-scope counts into a unique-repeater total: memberships overlap. MeshMapper's enabled/public population also differs from its boundary-based onboarding leaderboard. -## Maintainer follow-up: optional public per-IATA import +This is a names-and-counts catalogue. It supplies no repeater identities, per-repeater evidence, transport keys or channel decryption keys. A group's `zones` list does not assign every returned name or count to every member region. `monitored` and `wardriving` describe MeshMapper's configuration; they do not prove traffic or forwarding in Beacon. -The maintainer intends to provide an open get_scopes endpoint listing known scope names per IATA. The route, payload and deployment are not published yet; do not guess or probe endpoints. The agreed behavior is optional automatic enrichment, with Beacon's manual list retained for scopes absent from the remote list. +Responses advertise a five-minute cache lifetime and a strong ETag. Conditional requests use `If-None-Match`; an unchanged catalogue returns HTTP 304 with no body. `generated_at` is excluded from the ETag, so keep the source generation time separately from the last successful local check. The documented errors are 404 `zone_not_found`, 429 `rate_limited` with `Retry-After` in seconds, and 503 `unavailable`. The limit is 60 requests per minute per IP; schedule conservatively alongside other consumers. -Draft config names: `meshmapper.scopes.enabled` (default false), `meshmapper.scopes.url` (explicit published HTTPS endpoint), and `meshmapper.scopes.refresh_interval` (default 1h). No API-key option is required for the proposed public endpoint. Confirm names against the existing config layout when the adapter is implemented. +Live check on 27 September: YOW returned HTTP 200, `zones: ["YOW"]`, seven named entries including two at zero repeaters, `Cache-Control: public, max-age=300`, and an ETag. A conditional request with that ETag returned HTTP 304 and an empty body. These are point-in-time checks, not fixed regional totals or a stability test. -Import only IATAs configured in Beacon's regions. Effective names are the case-sensitive, deduplicated union of manual names and imported names for those IATAs; region unions must not turn an IATA-specific association into a global forwarding claim. Keep provenance and last successful synchronization time. A timeout, non-2xx, malformed/oversize response or unsupported schema retains the last-known-good imported catalogue and every manual scope, with an operator-visible stale state. A valid later snapshot may replace only that source's imported entries; never erase manual entries or independently observed evidence. Newly observed but unlisted scope identifiers remain explicitly unknown until named; do not invent names or overwrite manual transport keys. +## Beacon implementation boundary -Suggested API handoff: versioned JSON with IATA and arrays of exact scope names, plus generated/updated time; public read-only access, bounded response size and conditional refresh support if available. IATA metadata sync and scope-name sync remain separate capabilities, so either can be enabled independently. Scope catalog membership is not proof a repeater forwards that scope; observed forwarding/default-scope/app-discovery evidence retains its separate timestamps and expiry. +Beacon already has manual `scopes`, transport-code matching, node default scopes, observer scope associations, packet filters and channel SQL scope joins. Reuse them. Source audit: server candidate `99e623c5`, `internal/config/seed.go`, `internal/scopestore/scopestore.go`, `internal/ingest/packet.go` and `db/scopes.go`. + +- Keep the existing manual list. Preserve source spelling and case; use Beacon's existing normalization/key derivation when registering candidate names, and test its prefix handling against a known transport-packet fixture. Plain names currently receive `#`; explicit `#` and `$` prefixes are retained. Imported names must not overwrite manual metadata or keys. +- Track imported catalogue membership separately from traffic evidence. A successful import must never create observer associations, node defaults, forwarding claims or packet counts. Match actual packet transport codes through the existing ingest path before tagging messages. An unknown identifier stays unknown until supported by that match. +- Start with explicitly mapped single-region sources for IATAs configured in Beacon. Require the response's `region` and single-entry `zones` to match the configured IATA. Reject group responses in this first importer; later group support must preserve group-level provenance without inventing per-region attribution. +- Deduplicate canonical candidate names across sources while retaining each source's membership. A fresh snapshot replaces only that source's imported memberships. Never delete scope identities referenced by retained packets, manual entries or independently observed evidence because a remote name disappears. +- A failed or invalid refresh keeps the complete last-known-good snapshot and every manual entry. Preserve that snapshot across a restart. Expose the source, source generation time, last successful check and refresh failure to operators. HTTP 304 is a successful check, not a missing or empty catalogue; a valid empty 200 snapshot is a distinct case. +- Bound response bytes, source count, name length and effective candidate count. The current transport matcher iterates candidate keys per transport packet, so measure CPU cost at the supported catalogue limit. Publish a complete validated snapshot atomically, without a remote request in the ingest path or a partial update after failure. + +Proposed configuration namespace remains `meshmapper.scopes`: `enabled` defaults to false, `sources` explicitly maps an IATA to a published HTTPS endpoint, and `refresh_interval` defaults to one hour with a five-minute minimum. These names are a design proposal, not shipped YAML. Replace the earlier single-URL proposal; multiple regions need explicit mappings. No API-key setting is needed. Keep IATA metadata synchronization independently configurable. + +Use bounded HTTP timeouts, cached ETags and delayed retries respecting `Retry-After`. Schedule multiple sources without request bursts. Never scrape map pages, discover unpublished endpoints, or forward credentials to this public service. + +## Small delivery steps + +1. **Optional catalogue import.** Implement configuration, validation, cached conditional refresh, durable source membership and last-known-good handling. Reuse the existing scope store and name derivation, preserve manual values, and refresh the effective matching catalogue safely. Add operator-visible synchronization status. Import counts remain labelled MeshMapper catalogue counts, separate from Beacon's own observations. +2. **Channel message scope tags.** Audit existing history/catch-up queries and live message events, then expose the matched packet scope consistently in REST and WebSocket payloads. Add English/French tags, filter context and an explicit unknown state. Importing names does not decrypt a channel or recover already-purged packets. Historical backfill is separate work, not an implicit full-table rescan. +3. **Evidence and regional views.** Present advertised defaults, observed forwarding and observer-reported scopes with their own timestamps. Add group catalogues only with honest group-level attribution. Per-repeater imports or app-discovery evidence require a separate published contract; this API cannot supply them. + +Do not copy screenshot retention values or change MeshMapper's settings. Existing packet/analytics policies, public admin/backup restrictions and owner-controlled release/cutover remain in force. + +## Acceptance + +Cover 200/304, empty and zero-count entries, overlapping sources, case/prefix preservation, changed generation time with unchanged ETag, 404/429/503, timeout, malformed/oversize responses, region/group mismatch, restart during outage, and manual/imported name collisions. A failed update cannot erase the prior valid catalogue or partially activate names. An absent remote name cannot erase historical or independently observed evidence. + +Prove packet matching with a known fixture, including ambiguous candidates; catalogue membership alone is insufficient. For channel tags, reconcile history, catch-up and live delivery and verify English/French desktop/phone behavior. Run bounded native Pi tests and compare ingest CPU before enabling a source. Keep all current review candidates in the preview composition; update its changelog/source and preserve rollback for each application change. + +Review feedback and existing issues remain first priority. This contract makes the scope follow-up implementable; it does not imply that the importer or tagging work has shipped. From 5750c75998403f048b96d6128305e0fa5864ea68 Mon Sep 17 00:00:00 2001 From: n30nex Date: Sun, 27 Sep 2026 18:26:53 -0400 Subject: [PATCH 28/69] docs(scopes): record tested importer and preview recovery --- CONTRIBUTOR_WORKFLOW.md | 4 ++-- ROADMAP.md | 26 ++++++++++++++++++-------- app_config/config.yaml.example | 17 +++++++++++++++++ app_documentation/mesh-scopes-plan.md | 8 ++++---- 4 files changed, 41 insertions(+), 14 deletions(-) diff --git a/CONTRIBUTOR_WORKFLOW.md b/CONTRIBUTOR_WORKFLOW.md index a40609d..045e258 100644 --- a/CONTRIBUTOR_WORKFLOW.md +++ b/CONTRIBUTOR_WORKFLOW.md @@ -16,9 +16,9 @@ A source conflict still needs review. The helper automates routine history movem ## Current integration example -The 27 September refresh moved #166 independently and #167 -> #169 together onto accepted server #170. Route evidence #172 then follows #169; its web consumer #85 follows #83, and navigation #87 follows #85. The Pi composition includes every current candidate. Refresh/Publish/Check handles this ancestry once; source conflicts still require review. A history-only change with an identical tree still needs no Pi rebuild. +The 27 September refresh moved #166 independently and #167 -> #169 together onto accepted server #170. Route evidence #172 follows #169 and MeshMapper scope import #174 follows #172; its web consumer #85 follows #83, and navigation #87 follows #85. The Pi composition includes every current candidate. Refresh/Publish/Check handles this ancestry once; source conflicts still require review. A history-only change with an identical tree still needs no Pi rebuild. -Current preview is server `99e623c5` / web `98f820d2`, validated natively (929 tests) and publicly. The route API must be deployed before its UI. Rollback retains the previous schema040 database plus previous web assets. Follow the [roadmap](ROADMAP.md) for current issues, evidence boundaries and the next navigation slice. +Current preview is server `eb99f752` / web `98f820d2`, with native PostgreSQL and exact-head CI/race/security checks. Every existing candidate is retained; only YOW scope import is enabled. Rollback restores original schema041 database `beacon_pre042_20260927` and its matching server/configuration/source. The frontend is unchanged. Follow the [roadmap](ROADMAP.md) for issues, evidence and the next channel-scope slice. ## Setup diff --git a/ROADMAP.md b/ROADMAP.md index 013a73d..5750d29 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -6,19 +6,29 @@ Refresh GitHub issues, PR feedback and branch state before starting a phase. Thi Release-check correction, 21 September UTC: the workflow now includes independent preview PRs in Status and Check, applies the same CI/head/fork/target requirements to them, and rechecks the prepared independent inputs before publication. This covers packet summaries #161 and map correction #61 without adding them to the ordered stacks. The backup CLI #160 retains its separate check. Twenty-three offline regressions cover these gates and the existing no-rebase/cache behavior. See [the contributor workflow](CONTRIBUTOR_WORKFLOW.md). +## Regional scope import — 27 September + +[Server #174](https://github.com/MeshCore-Beacon/beacon-server/pull/174), `71e4e871`, follows #172 and closes focused issue #173 on acceptance. It imports public MeshMapper catalogue names into the existing matcher, keeps manual settings, persists last-known-good data and retry timing, and reports synchronization status in operator logs. Regional candidate limits and short-code ambiguity checks prevent a catalogue entry from becoming a forwarding claim. The feature defaults off; channel tags remain a separate slice. + +The Pi runs composed server `eb99f7523727b7e4208667e201f50ad235ce2c5f` with unchanged web `98f820d2`. Every prior review candidate remains included. Its explicit Ottawa/YOW source imported seven names; the public filter and actual incoming `#yow` adverts were verified. Native PostgreSQL tests and published-head CI/race/security checks pass. The maximum-catalogue matcher fixture took about 0.15ms per transport packet on the Pi; this is a microbenchmark, not a production-capacity claim. Pi ThreadSanitizer cannot run with the host's address layout, so race validation is on Linux CI. + +Migration 042 preserved all 31 existing table fingerprints in a restored copy. Immediate rollback retains original schema041 database `beacon_pre042_20260927` and backup `scopes-cutover-20260927T221330Z`; a checksum-verified private dump is also off the Pi. Only Beacon restarted, 22 other containers were unchanged, and both feeds reconnected. The [changelog and source](https://canadaverse.org/beacon-dev/source.html), 18 web assets and English/French-mode desktop/phone scope filtering were checked. Existing untranslated packet controls remain under #12; web source is unchanged. Retention remains 72h/30d/720h, and public admin/backup/foreign detection stay disabled. + +**Next:** review feedback and listed issues first, then channel message scope tags using the stored packet evidence across history, catch-up and live delivery. Group catalogues and regional evidence UI follow separately. Maintainers retain merges, stable releases and production cutover. + ## Observer investigation navigation — 27 September [Web #87](https://github.com/MeshCore-Beacon/beacon-web/pull/87), final candidate `98f820d2b2af5a69faf3f9aaf30d5f14b45feea8`, follows #85 and closes focused issue #86 on acceptance. Escape/Close dismisses the active panel and restores focus. Observer adverts are keyboard buttons and select the exact packet observation; embedded packet inspection preserves its originating URL. Revisiting an open entity returns to its existing panel, while tab/region/entity changes discard obsolete panels. Opening an observer dashboard keeps one originating screen mounted under its original Router location. Period, picker, comparison and directory detours stay within that visit; Back or the translated return action restores route/packet/node/map/Analytics state. A real route retained its typed filter, sort and 720px scroll position; a panned map's copied centre/zoom/layer/node link was identical after return. The Analytics leaderboard uses the same handler and provides keyboard buttons beside the canvas, using existing data. Direct/copied/reloaded dashboards remain standalone; arbitrary in-memory state is not persisted across reload. -The Pi now serves web `98f820d2b2af5a69faf3f9aaf30d5f14b45feea8` on unchanged server `99e623c5`. Final native build/lint and **929 tests in 106 files** pass; exact-head CI passes (web CodeQL remains skipped). All 18 public assets and both source archives match. Public LIVE, both MQTT feeds and return/keyboard journeys pass. A real packet-list regression is fixed: the retained origin stays invisible/inert with its layout intact. Final public checks preserve 442px scroll and 506px viewport height throughout the visit, plus the selected report URL. No containers restarted. Immediate web rollback is `5a261162` at `web-20260927T210108Z`; the phase-start `3b3abdcc` recovery remains at `web-20260927T202233Z`. Existing database recovery and 72h/30d/720h retention policies are unchanged. All prior review candidates remain included. +At the observer-navigation checkpoint, the Pi served web `98f820d2b2af5a69faf3f9aaf30d5f14b45feea8` on server `99e623c5`. Final native build/lint and **929 tests in 106 files** pass; exact-head CI passes (web CodeQL remains skipped). All 18 public assets and both source archives match. Public LIVE, both MQTT feeds and return/keyboard journeys pass. A real packet-list regression is fixed: the retained origin stays invisible/inert with its layout intact. Final public checks preserve 442px scroll and 506px viewport height throughout the visit, plus the selected report URL. No containers restarted. Immediate web rollback is `5a261162` at `web-20260927T210108Z`; the phase-start `3b3abdcc` recovery remains at `web-20260927T202233Z`. Existing database recovery and 72h/30d/720h retention policies are unchanged. All prior review candidates remain included. -**Next:** review feedback and listed issues first, then the next focused web #12 translation slice for the observer directory and quick-detail panels. Node/trace presentation and further reach/timing analytics follow the approved roadmap. The public MeshMapper scopes contract is now verified; optional catalogue import is implementable as a separate follow-up, with manual scopes retained. Maintainers control merges, stable releases and production cutover. +**Observer follow-up:** web #12 still covers the observer directory and quick-detail translations. Node/trace presentation and further reach/timing analytics follow the approved roadmap. Scope import #174 is implemented above; channel tags are its next separate slice. Maintainers control merges, stable releases and production cutover. ## MeshMapper scopes contract — 27 September -The [public API](https://wiki.meshmapper.net/scopes-api/) is available. The documented YOW endpoint returned HTTP 200 and a successful conditional HTTP 304; it requires no API key. The [scope integration plan](app_documentation/mesh-scopes-plan.md) now specifies explicit per-IATA sources, cached refresh, durable last-known-good data, manual-name preservation and separate imported/observed evidence. Group results cannot be attributed to individual member IATAs. The former unpublished-endpoint blocker is removed; the importer and channel tags remain unimplemented. Review feedback and listed issues retain priority. This documentation update changes no application revision, Pi service or review-stack dependency. +The [public API](https://wiki.meshmapper.net/scopes-api/) is available. The documented YOW endpoint returned HTTP 200 and a successful conditional HTTP 304; it requires no API key. The [scope integration plan](app_documentation/mesh-scopes-plan.md) now specifies explicit per-IATA sources, cached refresh, durable last-known-good data, manual-name preservation and separate imported/observed evidence. Group results cannot be attributed to individual member IATAs. The former unpublished-endpoint blocker is removed; importer #174 is now deployed for review, and channel tags remain the next separate slice. Review feedback and listed issues retain priority. That contract-only update preceded the tested importer deployment recorded above. ## Saved-route evidence — 27 September @@ -32,7 +42,7 @@ Current preview backend is composed `99e623c56477fd9b667d5f56bfb1a0eff34ecb2b`. At the route-evidence checkpoint, the Pi served web `3b3abdcc5bfa85b60c8959715736ea42c3d0bbd8`. The final native build/lint and all **915 tests** pass; exact-head CI passes (web CodeQL remains skipped). Desktop and 390px phone, English/French, copied/shared links, Back, keyboard Close/focus, exact report/node/observer and selected-map journeys were verified. All 18 public assets and both source archives match the tested artifacts. The public page is LIVE; both MQTT feeds are connected. Frontend publication restarted no containers. Web rollback retains `1d5d65e` in `web-20260927T192154Z`; the [source/changelog](https://canadaverse.org/beacon-dev/source.html) lists the complete review composition. -**Follow-up implemented above:** [web #86](https://github.com/MeshCore-Beacon/beacon-web/issues/86): observer quick-inspection Escape handling and broader cross-tab/overlay return navigation. That earlier Escape limitation is addressed by #87; acceptance remains with the maintainer. Node/trace evidence and additional reach/timing analytics follow that connection work. MeshMapper has now published its scopes contract; the separate import plan is updated, but the integration is not yet implemented. Broad server #60/#72/#99/#116 and web #12 remain open; this phase does not claim full parity or production capacity. Maintainers retain merges, stable releases and production cutover. +**Follow-up implemented above:** [web #86](https://github.com/MeshCore-Beacon/beacon-web/issues/86): observer quick-inspection Escape handling and broader cross-tab/overlay return navigation. That earlier Escape limitation is addressed by #87; acceptance remains with the maintainer. Node/trace evidence and additional reach/timing analytics follow that connection work. MeshMapper scope import #174 is now implemented and recorded above; its later channel/UI slices remain open. Broad server #60/#72/#99/#116 and web #12 remain open; this phase does not claim full parity or production capacity. Maintainers retain merges, stable releases and production cutover. ## Packet reception investigation — 27 September @@ -42,7 +52,7 @@ Map projection omits ambiguous/unlocated identities and breaks lines at gaps. Li At the packet-investigation checkpoint, the Pi ran web `1d5d65e` with server `88c2c10c`. Native build/lint and all **906 tests** pass; focused Windows checks and desktop/390px phone/English/French/keyboard/Back/shared-link checks pass. Public assets and source match, both MQTT feeds are connected, and the frontend publication restarted no services. Earlier review candidates remain included. The [changelog/source](https://canadaverse.org/beacon-dev/source.html) identifies the running build. Maintainers still own merges, stable releases and production cutover. -**Follow-up:** the saved-route evidence phase above implements this API and interface. Remaining work is observer return navigation. Non-packet overlay return navigation remains a separate follow-up. The separate MeshMapper scope plan now uses the published and verified API; implementation remains queued. Broader server #60/#72/#99/#116 and web #12 remain open; this is a first connected-investigation slice, not full parity. +**Follow-up:** the saved-route evidence phase above implements this API and interface. Remaining work is observer return navigation. Non-packet overlay return navigation remains a separate follow-up. The separate MeshMapper scope plan now has importer #174 implemented; channel tags remain queued. Broader server #60/#72/#99/#116 and web #12 remain open; this is a first connected-investigation slice, not full parity. ## Direction @@ -140,10 +150,10 @@ The September 24 consolidation check built accepted server `c02317a4` and retain The September 20 #116 investigation has a new [current-build result](https://github.com/MeshCore-Beacon/beacon-server/issues/116#issuecomment-5753626821): a 600-second unmodified Pi capture kept both feeds connected and retained 2,169 new observations, with no ping timeout, disconnect, deadline, SQLSTATE error or HTTP 5xx response. App/PostgreSQL CPU averaged 2.14%/3.96% of one core. The preceding 3h39 log likewise has no MQTT loss or deadline error. Timestamp warnings were classified separately. This did not measure callback or pool-acquisition duration and does not establish the original cause or production capacity. No application, ordering, acknowledgement or service change was made; #116 remains open. Further capture should follow a recurrence or meaningful workload change, rather than repeatedly sampling the same healthy state. -1. **Review the current retention/endpoint and observer candidates.** Keep the ordered server #167 -> #169 and web #75 -> #79 -> #80 -> #81 stacks; #166 is independent. Refresh with the existing workflow after acceptance. Maintainers choose the release breakpoint, versions, tags and main promotion. +1. **Review the current retention/endpoint and observer candidates.** Keep the ordered server #167 -> #169 -> #172 -> #174 and web #75 -> #79 -> #80 -> #81 -> #83 -> #85 -> #87 stacks; #166 is independent. Refresh with the existing workflow after acceptance. Maintainers choose the release breakpoint, versions, tags and main promotion. 2. **Connected investigation.** Connect packets, exact observed paths/routes, reporting observers and map actions with reliable Back navigation and visibly ambiguous identities. Continue focused issue #99/#12 work where it overlaps this accepted scope. 3. **Node/route/trace presentation, then distinct analytics questions and quality of life.** Follow the approved observer plan's subsequent releases; this phase does not claim full parity. -4. **Mesh Scopes interoperability.** The published regional API is verified; implement the optional cached importer, then consistent channel scope tags, using the [integration plan](app_documentation/mesh-scopes-plan.md). Retain manual names and separate observed/default/imported evidence. No API key is required; the importer is not yet shipped. +4. **Mesh Scopes interoperability.** Optional cached import is implemented in server #174; continue with consistent channel scope tags, using the [integration plan](app_documentation/mesh-scopes-plan.md). Retain manual names and separate observed/default/imported evidence. No API key is required; importer acceptance and release remain with maintainers. ## Listed work still open @@ -192,7 +202,7 @@ Matching tab names is not acceptance. Each capability needs verified semantics, - Verify release artifacts from reviewed source and applicable CI. The upstream web CodeQL workflow is currently disabled; its skipped job does not count as a security scan. - Publish matching source, configuration guidance, known limitations and a verified rollback procedure. The deployment owner performs the production switch. -The development preview still has limited accumulated history; configured 30-day retention does not mean a measured month is available. MeshMapper catalogue import has not been enabled. Its public admin/backup and foreign detection are disabled. These limitations remain explicit until configuration and validation support enabling them. +The development preview still has limited accumulated history; configured 30-day retention does not mean a measured month is available. MeshMapper catalogue import is enabled only for the explicitly configured YOW preview source. Its public admin/backup and foreign detection are disabled. These limitations remain explicit until configuration and validation support enabling them. ## Keeping this roadmap useful diff --git a/app_config/config.yaml.example b/app_config/config.yaml.example index 45618e0..c221362 100644 --- a/app_config/config.yaml.example +++ b/app_config/config.yaml.example @@ -105,3 +105,20 @@ cache: # allow_countries: [CA, US] # allow_continents: [NA] + +# Scope discovery below requires the server #174 review candidate until merged. +# Optional public MeshMapper scope-name discovery. Manual scopes remain authoritative. +# No API key is needed. Sources must belong to a configured region's IATAs. +# Group endpoints are not supported: their names cannot be attributed to each IATA. +# At most 16 sources, 64 names per source, 64 KiB per response; no page scraping. +# Catalogues and ETags persist in PostgreSQL. Failed refreshes retain known names. +# Refresh success/failure/freshness is logged under component=meshmapper.scopes. +# Removing/turning off a source deactivates its imported matching keys on restart; +# historical scope identities and recorded evidence are retained. +#meshmapper: +# scopes: +# enabled: false +# refresh_interval: 1h # 5m-24h; one source checked per 15s tick +# sources: +# YOW: https://yow.meshmapper.net/get_scopes.php + diff --git a/app_documentation/mesh-scopes-plan.md b/app_documentation/mesh-scopes-plan.md index f30c587..2c35202 100644 --- a/app_documentation/mesh-scopes-plan.md +++ b/app_documentation/mesh-scopes-plan.md @@ -1,6 +1,6 @@ # Mesh Scopes interoperability plan -Updated 27 September 2026 UTC. The [MeshMapper Scopes API](https://wiki.meshmapper.net/scopes-api/) is published and its regional endpoint and conditional caching were checked live. The API-contract blocker is removed. Beacon's importer is still unimplemented and disabled; this document specifies the next focused scope work, not an available configuration feature. +Updated 27 September 2026 UTC. The [MeshMapper Scopes API](https://wiki.meshmapper.net/scopes-api/) is published and its regional endpoint and conditional caching were checked live. The API-contract blocker is removed. The first importer is implemented in [server PR #174](https://github.com/MeshCore-Beacon/beacon-server/pull/174), following #172 and closing issue #173 on acceptance. It defaults off; channel scope tags remain the next separate slice. Maintainers retain acceptance and release. ## Published contract @@ -25,13 +25,13 @@ Beacon already has manual `scopes`, transport-code matching, node default scopes - A failed or invalid refresh keeps the complete last-known-good snapshot and every manual entry. Preserve that snapshot across a restart. Expose the source, source generation time, last successful check and refresh failure to operators. HTTP 304 is a successful check, not a missing or empty catalogue; a valid empty 200 snapshot is a distinct case. - Bound response bytes, source count, name length and effective candidate count. The current transport matcher iterates candidate keys per transport packet, so measure CPU cost at the supported catalogue limit. Publish a complete validated snapshot atomically, without a remote request in the ingest path or a partial update after failure. -Proposed configuration namespace remains `meshmapper.scopes`: `enabled` defaults to false, `sources` explicitly maps an IATA to a published HTTPS endpoint, and `refresh_interval` defaults to one hour with a five-minute minimum. These names are a design proposal, not shipped YAML. Replace the earlier single-URL proposal; multiple regions need explicit mappings. No API-key setting is needed. Keep IATA metadata synchronization independently configurable. +The review candidate implements `meshmapper.scopes.enabled` (default false), `sources` (explicit IATA-to-URL map), and `refresh_interval` (default 1h, allowed 5m-24h). At most 16 sources are polled one per 15-second tick; each response is limited to 64 KiB and 64 names. Only published HTTPS MeshMapper regional endpoints are accepted; redirects and group responses are rejected. Operator logs report successful check/source times and refresh errors. IATA metadata synchronization remains separate; no API key is needed. See the [server configuration and recovery notes](https://github.com/n30nex/beacon-server-contributions/blob/codex/beacon-meshmapper-scopes/docs/meshmapper-scopes.md). Use bounded HTTP timeouts, cached ETags and delayed retries respecting `Retry-After`. Schedule multiple sources without request bursts. Never scrape map pages, discover unpublished endpoints, or forward credentials to this public service. ## Small delivery steps -1. **Optional catalogue import.** Implement configuration, validation, cached conditional refresh, durable source membership and last-known-good handling. Reuse the existing scope store and name derivation, preserve manual values, and refresh the effective matching catalogue safely. Add operator-visible synchronization status. Import counts remain labelled MeshMapper catalogue counts, separate from Beacon's own observations. +1. **Optional catalogue import — implemented in #174, awaiting acceptance.** Configuration, validation, cached conditional refresh, durable source membership and last-known-good handling are included. Reuse the existing scope store and name derivation, preserve manual values, and refresh the effective matching catalogue safely. Add operator-visible synchronization status. Import counts remain labelled MeshMapper catalogue counts, separate from Beacon's own observations. 2. **Channel message scope tags.** Audit existing history/catch-up queries and live message events, then expose the matched packet scope consistently in REST and WebSocket payloads. Add English/French tags, filter context and an explicit unknown state. Importing names does not decrypt a channel or recover already-purged packets. Historical backfill is separate work, not an implicit full-table rescan. 3. **Evidence and regional views.** Present advertised defaults, observed forwarding and observer-reported scopes with their own timestamps. Add group catalogues only with honest group-level attribution. Per-repeater imports or app-discovery evidence require a separate published contract; this API cannot supply them. @@ -43,4 +43,4 @@ Cover 200/304, empty and zero-count entries, overlapping sources, case/prefix pr Prove packet matching with a known fixture, including ambiguous candidates; catalogue membership alone is insufficient. For channel tags, reconcile history, catch-up and live delivery and verify English/French desktop/phone behavior. Run bounded native Pi tests and compare ingest CPU before enabling a source. Keep all current review candidates in the preview composition; update its changelog/source and preserve rollback for each application change. -Review feedback and existing issues remain first priority. This contract makes the scope follow-up implementable; it does not imply that the importer or tagging work has shipped. +Review feedback and existing issues remain first priority. The importer is a review candidate; public channel tags, group catalogues and regional evidence UI are not implemented by #174. See the roadmap for the exact tested preview revision and retained rollback. From e2bcfcd294cf99d18910453db5bfbe72b3b064c1 Mon Sep 17 00:00:00 2001 From: n30nex Date: Sun, 27 Sep 2026 20:05:56 -0400 Subject: [PATCH 29/69] docs: record channel scope delivery and boundary follow-up --- ROADMAP.md | 22 ++++++++++++++----- app_documentation/mesh-scopes-plan.md | 6 ++--- .../meshmapper-boundaries-plan.md | 11 ++++++++++ 3 files changed, 31 insertions(+), 8 deletions(-) create mode 100644 app_documentation/meshmapper-boundaries-plan.md diff --git a/ROADMAP.md b/ROADMAP.md index 5750d29..7a90668 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -6,15 +6,27 @@ Refresh GitHub issues, PR feedback and branch state before starting a phase. Thi Release-check correction, 21 September UTC: the workflow now includes independent preview PRs in Status and Check, applies the same CI/head/fork/target requirements to them, and rechecks the prepared independent inputs before publication. This covers packet summaries #161 and map correction #61 without adding them to the ordered stacks. The backup CLI #160 retains its separate check. Twenty-three offline regressions cover these gates and the existing no-rebase/cache behavior. See [the contributor workflow](CONTRIBUTOR_WORKFLOW.md). +## Channel scope investigation and Public channel — 27 September + +[Server #176](https://github.com/MeshCore-Beacon/beacon-server/pull/176) (`7fc631e8`, follows #174, closes #175) and [web #89](https://github.com/MeshCore-Beacon/beacon-web/pull/89) (`e7618fd7`, follows #87, closes #88) expose first-recorded packet scope consistently in history, catch-up and live messages. The interface adds scope filtering, packet inspection, distinct unknown/unavailable/unscoped states and English/French explanations. Duplicate broker messages and live arrivals during a history request preserve the existing page cursor and message counts. Imported catalogue names alone are not forwarding evidence. + +The current Pi preview is composed server `7c9599b167bcad2b416ef03304ff27909ab3021b` / web `e7618fd7d40aff4e01f581999fdbb21d10de2e67`, with every earlier review candidate included. Native server/PostgreSQL tests, all 935 web tests, build/lint and published-head CI pass (web CodeQL is skipped). Browser checks cover live/history filtering, keyboard inspection, English/French and a 390px phone. The public page is LIVE, both MQTT inputs are connected, and all 18 assets plus both source archives match the [changelog/source offer](https://canadaverse.org/beacon-dev/source.html). + +The standard MeshCore Public channel key is enabled at the user's request, matching the known non-hashtag hash-11 channel on dev.meshcore.ca. A restored-copy trial recovered 2,735 retained messages in 16.127 seconds; public decoded history and a new incoming message were verified. Expired packets remain unavailable. Schema042 is unchanged. Rollback retains server `eb99f752`, web `98f820d2`, configuration and source without discarding new database rows; the older schema041 recovery remains separately available. Packets stay 72h, summaries 30d and telemetry 720h. Public admin, backups and foreign detection remain disabled. + +See the [boundary integration plan](app_documentation/meshmapper-boundaries-plan.md). + +**Next:** refresh review feedback and listed issues first, then optional MeshMapper boundary synchronization using the published Zones API and existing map layer. Preserve manual boundaries and cached geometry; keep cross-boundary packet/route investigation in a later focused contribution. Group scope catalogues, regional evidence, existing translations and broader parity work remain open. Maintainers control acceptance, merge order and production release. + ## Regional scope import — 27 September [Server #174](https://github.com/MeshCore-Beacon/beacon-server/pull/174), `71e4e871`, follows #172 and closes focused issue #173 on acceptance. It imports public MeshMapper catalogue names into the existing matcher, keeps manual settings, persists last-known-good data and retry timing, and reports synchronization status in operator logs. Regional candidate limits and short-code ambiguity checks prevent a catalogue entry from becoming a forwarding claim. The feature defaults off; channel tags remain a separate slice. -The Pi runs composed server `eb99f7523727b7e4208667e201f50ad235ce2c5f` with unchanged web `98f820d2`. Every prior review candidate remains included. Its explicit Ottawa/YOW source imported seven names; the public filter and actual incoming `#yow` adverts were verified. Native PostgreSQL tests and published-head CI/race/security checks pass. The maximum-catalogue matcher fixture took about 0.15ms per transport packet on the Pi; this is a microbenchmark, not a production-capacity claim. Pi ThreadSanitizer cannot run with the host's address layout, so race validation is on Linux CI. +At the scope-import checkpoint, the Pi ran composed server `eb99f7523727b7e4208667e201f50ad235ce2c5f` with unchanged web `98f820d2`. Every prior review candidate remains included. Its explicit Ottawa/YOW source imported seven names; the public filter and actual incoming `#yow` adverts were verified. Native PostgreSQL tests and published-head CI/race/security checks pass. The maximum-catalogue matcher fixture took about 0.15ms per transport packet on the Pi; this is a microbenchmark, not a production-capacity claim. Pi ThreadSanitizer cannot run with the host's address layout, so race validation is on Linux CI. Migration 042 preserved all 31 existing table fingerprints in a restored copy. Immediate rollback retains original schema041 database `beacon_pre042_20260927` and backup `scopes-cutover-20260927T221330Z`; a checksum-verified private dump is also off the Pi. Only Beacon restarted, 22 other containers were unchanged, and both feeds reconnected. The [changelog and source](https://canadaverse.org/beacon-dev/source.html), 18 web assets and English/French-mode desktop/phone scope filtering were checked. Existing untranslated packet controls remain under #12; web source is unchanged. Retention remains 72h/30d/720h, and public admin/backup/foreign detection stay disabled. -**Next:** review feedback and listed issues first, then channel message scope tags using the stored packet evidence across history, catch-up and live delivery. Group catalogues and regional evidence UI follow separately. Maintainers retain merges, stable releases and production cutover. +**Follow-up delivered above:** channel tags are in #176/#89. Group catalogues and regional evidence remain separate future slices. Maintainers retain merges, stable releases and production cutover. ## Observer investigation navigation — 27 September @@ -24,11 +36,11 @@ Opening an observer dashboard keeps one originating screen mounted under its ori At the observer-navigation checkpoint, the Pi served web `98f820d2b2af5a69faf3f9aaf30d5f14b45feea8` on server `99e623c5`. Final native build/lint and **929 tests in 106 files** pass; exact-head CI passes (web CodeQL remains skipped). All 18 public assets and both source archives match. Public LIVE, both MQTT feeds and return/keyboard journeys pass. A real packet-list regression is fixed: the retained origin stays invisible/inert with its layout intact. Final public checks preserve 442px scroll and 506px viewport height throughout the visit, plus the selected report URL. No containers restarted. Immediate web rollback is `5a261162` at `web-20260927T210108Z`; the phase-start `3b3abdcc` recovery remains at `web-20260927T202233Z`. Existing database recovery and 72h/30d/720h retention policies are unchanged. All prior review candidates remain included. -**Observer follow-up:** web #12 still covers the observer directory and quick-detail translations. Node/trace presentation and further reach/timing analytics follow the approved roadmap. Scope import #174 is implemented above; channel tags are its next separate slice. Maintainers control merges, stable releases and production cutover. +**Observer follow-up:** web #12 still covers the observer directory and quick-detail translations. Node/trace presentation and further reach/timing analytics follow the approved roadmap. Scope import #174 and channel tags #176/#89 are implemented above. Maintainers control merges, stable releases and production cutover. ## MeshMapper scopes contract — 27 September -The [public API](https://wiki.meshmapper.net/scopes-api/) is available. The documented YOW endpoint returned HTTP 200 and a successful conditional HTTP 304; it requires no API key. The [scope integration plan](app_documentation/mesh-scopes-plan.md) now specifies explicit per-IATA sources, cached refresh, durable last-known-good data, manual-name preservation and separate imported/observed evidence. Group results cannot be attributed to individual member IATAs. The former unpublished-endpoint blocker is removed; importer #174 is now deployed for review, and channel tags remain the next separate slice. Review feedback and listed issues retain priority. That contract-only update preceded the tested importer deployment recorded above. +The [public API](https://wiki.meshmapper.net/scopes-api/) is available. The documented YOW endpoint returned HTTP 200 and a successful conditional HTTP 304; it requires no API key. The [scope integration plan](app_documentation/mesh-scopes-plan.md) now specifies explicit per-IATA sources, cached refresh, durable last-known-good data, manual-name preservation and separate imported/observed evidence. Group results cannot be attributed to individual member IATAs. The former unpublished-endpoint blocker is removed; importer #174 and channel tags #176/#89 are now deployed for review. Review feedback and listed issues retain priority. That contract-only update preceded the tested importer deployment recorded above. ## Saved-route evidence — 27 September @@ -52,7 +64,7 @@ Map projection omits ambiguous/unlocated identities and breaks lines at gaps. Li At the packet-investigation checkpoint, the Pi ran web `1d5d65e` with server `88c2c10c`. Native build/lint and all **906 tests** pass; focused Windows checks and desktop/390px phone/English/French/keyboard/Back/shared-link checks pass. Public assets and source match, both MQTT feeds are connected, and the frontend publication restarted no services. Earlier review candidates remain included. The [changelog/source](https://canadaverse.org/beacon-dev/source.html) identifies the running build. Maintainers still own merges, stable releases and production cutover. -**Follow-up:** the saved-route evidence phase above implements this API and interface. Remaining work is observer return navigation. Non-packet overlay return navigation remains a separate follow-up. The separate MeshMapper scope plan now has importer #174 implemented; channel tags remain queued. Broader server #60/#72/#99/#116 and web #12 remain open; this is a first connected-investigation slice, not full parity. +**Follow-up:** the saved-route evidence phase above implements this API and interface. Remaining work is observer return navigation. Non-packet overlay return navigation remains a separate follow-up. The separate MeshMapper scope plan now has importer #174 and channel tags #176/#89 implemented. Broader server #60/#72/#99/#116 and web #12 remain open; this is a first connected-investigation slice, not full parity. ## Direction diff --git a/app_documentation/mesh-scopes-plan.md b/app_documentation/mesh-scopes-plan.md index 2c35202..36a2976 100644 --- a/app_documentation/mesh-scopes-plan.md +++ b/app_documentation/mesh-scopes-plan.md @@ -1,6 +1,6 @@ # Mesh Scopes interoperability plan -Updated 27 September 2026 UTC. The [MeshMapper Scopes API](https://wiki.meshmapper.net/scopes-api/) is published and its regional endpoint and conditional caching were checked live. The API-contract blocker is removed. The first importer is implemented in [server PR #174](https://github.com/MeshCore-Beacon/beacon-server/pull/174), following #172 and closing issue #173 on acceptance. It defaults off; channel scope tags remain the next separate slice. Maintainers retain acceptance and release. +Updated 27 September 2026 UTC. The [MeshMapper Scopes API](https://wiki.meshmapper.net/scopes-api/) is published and its regional endpoint and conditional caching were checked live. The API-contract blocker is removed. The first importer is implemented in [server PR #174](https://github.com/MeshCore-Beacon/beacon-server/pull/174), following #172 and closing issue #173 on acceptance. It defaults off; channel tags are implemented in server #176 / web #89, awaiting acceptance. Maintainers retain acceptance and release. ## Published contract @@ -32,7 +32,7 @@ Use bounded HTTP timeouts, cached ETags and delayed retries respecting `Retry-Af ## Small delivery steps 1. **Optional catalogue import — implemented in #174, awaiting acceptance.** Configuration, validation, cached conditional refresh, durable source membership and last-known-good handling are included. Reuse the existing scope store and name derivation, preserve manual values, and refresh the effective matching catalogue safely. Add operator-visible synchronization status. Import counts remain labelled MeshMapper catalogue counts, separate from Beacon's own observations. -2. **Channel message scope tags.** Audit existing history/catch-up queries and live message events, then expose the matched packet scope consistently in REST and WebSocket payloads. Add English/French tags, filter context and an explicit unknown state. Importing names does not decrypt a channel or recover already-purged packets. Historical backfill is separate work, not an implicit full-table rescan. +2. **Channel message scope tags — implemented in server #176 / web #89, awaiting acceptance.** REST history, hash/global lists, catch-up and live events share first-stored-packet metadata. English/French tags, retained-history/live filtering and distinct unscoped, unresolved and unavailable states are included. A later reception does not silently rewrite the first packet evidence. Importing names does not decrypt a channel or recover already-purged packets. Separately, the user enabled the standard Public key on the preview after a restored-copy backfill trial; private keys remain unconfigured. 3. **Evidence and regional views.** Present advertised defaults, observed forwarding and observer-reported scopes with their own timestamps. Add group catalogues only with honest group-level attribution. Per-repeater imports or app-discovery evidence require a separate published contract; this API cannot supply them. Do not copy screenshot retention values or change MeshMapper's settings. Existing packet/analytics policies, public admin/backup restrictions and owner-controlled release/cutover remain in force. @@ -43,4 +43,4 @@ Cover 200/304, empty and zero-count entries, overlapping sources, case/prefix pr Prove packet matching with a known fixture, including ambiguous candidates; catalogue membership alone is insufficient. For channel tags, reconcile history, catch-up and live delivery and verify English/French desktop/phone behavior. Run bounded native Pi tests and compare ingest CPU before enabling a source. Keep all current review candidates in the preview composition; update its changelog/source and preserve rollback for each application change. -Review feedback and existing issues remain first priority. The importer is a review candidate; public channel tags, group catalogues and regional evidence UI are not implemented by #174. See the roadmap for the exact tested preview revision and retained rollback. +Review feedback and existing issues remain first priority. The importer is a review candidate; channel tags are delivered by #176/#89; group catalogues and regional evidence UI remain future work. See the roadmap for the exact tested preview revision and retained rollback. diff --git a/app_documentation/meshmapper-boundaries-plan.md b/app_documentation/meshmapper-boundaries-plan.md new file mode 100644 index 0000000..61ef3c1 --- /dev/null +++ b/app_documentation/meshmapper-boundaries-plan.md @@ -0,0 +1,11 @@ +# Optional MeshMapper boundary synchronization + +Queued after the channel-scope slice, from the maintainer discussion supplied on 27 September. The [published Zones API](https://wiki.meshmapper.net/zones-api/) already supplies the required catalogue and polygons; no scraping or new endpoint is needed. + +Verified public reads: `https://meshmapper.net/get_zones.php?country=CA` lists enabled regions, and `https://yow.meshmapper.net/get_geojson.php` returns YOW's FeatureCollection. Both are unauthenticated and support ETag caching with a one-hour minimum polling period. Coordinates use longitude, latitude. A missing boundary is explicitly null; group collections contain separate member features. Region codes can be 2–6 alphanumeric characters, so check compatibility with Beacon identifiers before importing or mapping them. Never truncate codes. + +Implement an independently enabled boundary source, preserving manual `border_file` precedence. Reuse Beacon's existing GeoJSON validator, border storage and map layer. Bound downloads and vertices, validate the configured region against each feature, preserve source/freshness information and last-known-good geometry, and respect conditional responses and retry timing. Removing a remote boundary must not erase a manual one or synthesize a radius circle. Public foreign-node classification remains disabled unless the owner changes that setting. + +Deliver boundary import/map display first; put packet/route crossing investigation in a later focused PR. Crossing views must distinguish observed positions from inferred paths and explain gaps, ambiguous hashes and stale locations. Relate scopes through recorded packet evidence, never through a scope name's spelling. The maintainer explicitly noted that `#ottawa` came from an individual discovered node; that name is not an authoritative city boundary or network-wide forwarding claim. + +Keep scope-name and boundary synchronization independently configurable. Validate native Pi behavior, outage recovery, manual precedence, group/null/invalid geometry and English/French desktop/phone display before preview publication. Preserve the existing review composition and rollback; owners retain production release. From c49b784154a82b83734d55a7c8cbbd548d12df1c Mon Sep 17 00:00:00 2001 From: n30nex Date: Sun, 27 Sep 2026 20:08:53 -0400 Subject: [PATCH 30/69] docs: distinguish new upstream origin change from tested preview --- ROADMAP.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/ROADMAP.md b/ROADMAP.md index 7a90668..ed2076a 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -6,6 +6,8 @@ Refresh GitHub issues, PR feedback and branch state before starting a phase. Thi Release-check correction, 21 September UTC: the workflow now includes independent preview PRs in Status and Check, applies the same CI/head/fork/target requirements to them, and rechecks the prepared independent inputs before publication. This covers packet summaries #161 and map correction #61 without adding them to the ordered stacks. The backup CLI #160 retains its separate check. Twenty-three offline regressions cover these gates and the existing no-rebase/cache behavior. See [the contributor workflow](CONTRIBUTOR_WORKFLOW.md). +Final upstream refresh note: server #177 merged as `90f9b506` during channel-phase verification. The published server candidates and Pi composition below remain based on `dec643a2`. Their exact-head CI passes, but the stack helper requires one batch refresh before the next server publication; the validated running artifacts have not been relabelled. Web Check is current. + ## Channel scope investigation and Public channel — 27 September [Server #176](https://github.com/MeshCore-Beacon/beacon-server/pull/176) (`7fc631e8`, follows #174, closes #175) and [web #89](https://github.com/MeshCore-Beacon/beacon-web/pull/89) (`e7618fd7`, follows #87, closes #88) expose first-recorded packet scope consistently in history, catch-up and live messages. The interface adds scope filtering, packet inspection, distinct unknown/unavailable/unscoped states and English/French explanations. Duplicate broker messages and live arrivals during a history request preserve the existing page cursor and message counts. Imported catalogue names alone are not forwarding evidence. From 9cba0ddf4784d8439e337f376d0ac9195e05dd23 Mon Sep 17 00:00:00 2001 From: n30nex Date: Sun, 27 Sep 2026 21:06:08 -0400 Subject: [PATCH 31/69] docs: record Canada-US scope coverage and packet layout fix --- ROADMAP.md | 16 +- .../north-american-scope-candidates.json | 1130 +++++++++++++++++ 2 files changed, 1144 insertions(+), 2 deletions(-) create mode 100644 app_documentation/north-american-scope-candidates.json diff --git a/ROADMAP.md b/ROADMAP.md index ed2076a..69fbf12 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,12 +1,24 @@ # Beacon parity and analytics roadmap -Updated 27 September 2026 UTC. This is the working roadmap for n30nex's ongoing contributions toward CoreScope feature parity. Maintainers decide acceptance and merge order; deployment owners handle the production switch. +Updated 28 September 2026 UTC. This is the working roadmap for n30nex's ongoing contributions toward CoreScope feature parity. Maintainers decide acceptance and merge order; deployment owners handle the production switch. Refresh GitHub issues, PR feedback and branch state before starting a phase. This document is a snapshot, and linked issues/PRs are the current source of truth. Release-check correction, 21 September UTC: the workflow now includes independent preview PRs in Status and Check, applies the same CI/head/fork/target requirements to them, and rechecks the prepared independent inputs before publication. This covers packet summaries #161 and map correction #61 without adding them to the ordered stacks. The backup CLI #160 retains its separate check. Twenty-three offline regressions cover these gates and the existing no-rebase/cache behavior. See [the contributor workflow](CONTRIBUTOR_WORKFLOW.md). -Final upstream refresh note: server #177 merged as `90f9b506` during channel-phase verification. The published server candidates and Pi composition below remain based on `dec643a2`. Their exact-head CI passes, but the stack helper requires one batch refresh before the next server publication; the validated running artifacts have not been relabelled. Web Check is current. +Latest upstream is server `876d9970` (#177/#178) and web `17f48fb9` (#91). The tested preview retains its exact integrated bases; both review stacks require one helper-managed refresh before the next publication. Do not rebase children independently or relabel existing binaries. + +## Canada/US scope coverage and packet layout — 28 September + +The preview now has **504 exact-case scope candidates**: 261 distinct names from all 237 published Canadian/US regional scope catalogues, plus lowercase IATA/group, province/state, district/territory and country fallbacks. It covers 244 currently known region codes, all 13 Canadian province/territory codes, all 50 US states, DC and five US territories, and includes `#ca`, `#can`, `#us`, `#usa` and `#na`. Counts overlap; do not add these categories together. Published mixed-case names are preserved because case changes the key. Named scopes are not geographic boundaries or evidence of repeater use. + +The reported “Test 4” packet uniquely matches `#ykf`; its original unresolved label remains unchanged. Fresh “Ykf test” and “This is scoped to ykf only” messages now resolve as `#ykf`. A read-only audit of 932 retained transport packets found 670 unique candidate matches, 255 without a known match and seven short-code collisions. Unknown custom names cannot be recovered from these codes alone. Observer/neighbor reports currently contained only the wildcard `*`, so they supplied no additional names. No historical labels were rewritten. The native Pi matcher passes the captured-packet regression and measured a median 0.55ms for a full 504-name scan; this is not a production-capacity claim. + +See the [candidate snapshot and provenance](app_documentation/north-american-scope-candidates.json). This is a recorded catalogue snapshot, with the existing automatic YOW importer still active. For subsequent scope work, refresh Canadian/US published catalogues and observed IATA/report names within API cache/rate limits, retain manual fallbacks and surface unresolved/ambiguous codes. Do not claim an undisclosed recurring all-region discovery service. Arbitrary private names still require a published catalogue or an explicit supplied/reported name. + +[Web #92](https://github.com/MeshCore-Beacon/beacon-web/pull/92), `dfeb2777`, follows #89 and closes #90. It gives the route label and scope separate lines within a 128px track, preserving 37px rows and exact scope case. Long tags remain inside phone cards. A real browser geometry check reproduced the spill before the fix and passed afterward, including the user's `BB2F2752` row. Desktop, 768px table, 390px phone, keyboard and English/French public checks pass. All 935 tests pass on Windows and the Pi, as does exact-head CI; existing warnings remain and web CodeQL is skipped. + +The Pi now serves web `dfeb277756b1a9b230d7e7e0f2d45d62713bf45b` with unchanged server `7c9599b1`, every prior candidate, and the updated [source/changelog](https://canadaverse.org/beacon-dev/source.html). All 18 assets and both source archives match. Both inputs and public LIVE are verified; frontend publication restarted no containers. Application merges/releases remain owner-controlled. New upstream server #177/#178 and web #91 require the next single helper-managed stack refresh; the running source is not relabelled as those newer heads. ## Channel scope investigation and Public channel — 27 September diff --git a/app_documentation/north-american-scope-candidates.json b/app_documentation/north-american-scope-candidates.json new file mode 100644 index 0000000..bb5973f --- /dev/null +++ b/app_documentation/north-american-scope-candidates.json @@ -0,0 +1,1130 @@ +{ + "manual_scope_names": [ + "#ATW", + "#CHA", + "#CLT", + "#CVG", + "#CWA", + "#EAU", + "#EWN", + "#GDW", + "#GRR", + "#GSO", + "#ILM-MM", + "#LSE", + "#MHK", + "#MKE", + "#MSN", + "#MSP", + "#ORD", + "#RFD", + "#US-EAST-1", + "#US-EAST-AL", + "#US-EAST-FL", + "#US-EAST-GA", + "#US-EAST-NC", + "#US-EAST-SC", + "#US-EAST-TN", + "#ab", + "#abe", + "#abi", + "#abq", + "#acy", + "#ags", + "#aiz", + "#ak", + "#al", + "#alb", + "#ama", + "#aoh", + "#apf", + "#ar", + "#as", + "#ast", + "#atl", + "#atw", + "#aus", + "#avl", + "#axn", + "#az", + "#azo", + "#baus", + "#bay-area", + "#bc", + "#bdl", + "#bend", + "#bfd", + "#bfl", + "#bgm", + "#bgr", + "#bhm", + "#bli", + "#bna", + "#bob", + "#boi", + "#bos", + "#brk", + "#broom", + "#buf", + "#bvs", + "#bw", + "#bwi", + "#c-or", + "#c-ut", + "#c-wa", + "#ca", + "#cae", + "#can", + "#capitolhill", + "#cda", + "#ced", + "#cez", + "#cgi", + "#cha", + "#chgo", + "#chih", + "#chs", + "#cle", + "#cls", + "#clt", + "#cmh", + "#cmi", + "#co", + "#coa", + "#coast-or", + "#con", + "#cos", + "#crw", + "#ct", + "#ct-ffc", + "#ct-rv", + "#cv", + "#cvg", + "#cvo", + "#cwa", + "#dab", + "#day", + "#dc", + "#de", + "#dec", + "#default", + "#den", + "#det", + "#dfw", + "#dij", + "#dvn", + "#e-id", + "#e-idregion", + "#east", + "#eat", + "#eau", + "#ege", + "#ei", + "#eid", + "#elm", + "#elp", + "#erc", + "#ercregion", + "#eug", + "#evv", + "#ewn", + "#far", + "#fat", + "#fca", + "#fl", + "#flo", + "#fnl", + "#fnt", + "#fpr", + "#frd", + "#fwa", + "#ga", + "#gc", + "#gc-la", + "#gc-la-msy", + "#gc-la-msy-mm", + "#gc-la-msy-sja", + "#gdw", + "#geg", + "#gjt", + "#golm", + "#gpt", + "#gpz", + "#grh", + "#grr", + "#gso", + "#gsp", + "#gtp", + "#gtr", + "#gu", + "#guf", + "#hb", + "#hg", + "#hi", + "#hlg", + "#hou", + "#howltest", + "#hts", + "#hunting-group", + "#hv", + "#hyr", + "#ia", + "#iad", + "#id", + "#ida", + "#idaho", + "#il", + "#ilm", + "#imw", + "#in", + "#ind", + "#inw", + "#jac", + "#jan", + "#jax", + "#keh", + "#kit", + "#kls", + "#ks", + "#ky", + "#la", + "#las", + "#lax", + "#lbb", + "#lch", + "#lcq", + "#lex", + "#lft", + "#lgu", + "#lmt", + "#lns", + "#local", + "#loz", + "#lru", + "#lse", + "#lwc", + "#ma", + "#mac", + "#mad", + "#maf", + "#maine", + "#manitowoc", + "#mb", + "#mbs", + "#mce", + "#mci", + "#md", + "#mdt", + "#me", + "#mem", + "#mfe", + "#mfr", + "#mgm", + "#mhk", + "#mi", + "#mke", + "#mlb", + "#mlu", + "#mmap", + "#mmh", + "#mmv", + "#mn", + "#mo", + "#mot", + "#mp", + "#ms", + "#msn", + "#msp", + "#msv", + "#msy", + "#mt", + "#mtj", + "#n-ut", + "#na", + "#nb", + "#nc", + "#ncmc", + "#nd", + "#ne", + "#nh", + "#nj", + "#nl", + "#nm", + "#northeast", + "#ns", + "#nt", + "#ntu", + "#nu", + "#nv", + "#nwtx", + "#ny", + "#nyc", + "#oak", + "#ocf", + "#ogd", + "#ogg", + "#oh", + "#ok", + "#okc", + "#olm", + "#oly", + "#oma", + "#on", + "#onp", + "#onqc", + "#or", + "#ord", + "#orl", + "#oth", + "#otk", + "#ottawa", + "#oxr", + "#pa", + "#pae", + "#pah", + "#pdx", + "#pe", + "#pgd", + "#pgv", + "#phx", + "#pia", + "#pid", + "#pih", + "#pit", + "#pkb", + "#pns", + "#pnw", + "#ppm", + "#pr", + "#prb", + "#prc", + "#psc", + "#psf", + "#psk", + "#psp", + "#pub", + "#puw", + "#pv", + "#pvd", + "#qc", + "#rdd", + "#rdm", + "#rdu", + "#rfd", + "#ri", + "#ric", + "#ririe", + "#rkd", + "#rks", + "#rld", + "#rno", + "#roa", + "#roc", + "#rst", + "#rsw", + "#rut", + "#rxe", + "#s-id", + "#s-ut", + "#san", + "#sat", + "#sav", + "#save", + "#sba", + "#sbn", + "#sbp", + "#sby", + "#sc", + "#sce", + "#sd", + "#sdf", + "#se-", + "#se-wa", + "#sea", + "#seattle", + "#setx", + "#sfo", + "#sgf", + "#sgu", + "#sjc", + "#sjt", + "#sk", + "#slc", + "#sle", + "#smf", + "#smn", + "#snoco", + "#socal", + "#southeast", + "#southseattle", + "#spi", + "#srq", + "#stc", + "#stg", + "#stl", + "#sun", + "#sun-prairie", + "#svc", + "#sw-id", + "#sw-wa", + "#swbc", + "#swoh", + "#syr", + "#tdo", + "#test", + "#test-1e", + "#test-b5", + "#test-cc", + "#texas", + "#tlh", + "#tn", + "#tpa", + "#tri", + "#ttd", + "#tul", + "#twf", + "#twn", + "#tworivers", + "#tx", + "#txk", + "#tys", + "#united-states", + "#us", + "#us-al", + "#us-ca", + "#us-cm", + "#us-cm-mm", + "#us-east", + "#us-east-1", + "#us-east-al", + "#us-east-fl", + "#us-east-ga", + "#us-east-nc", + "#us-east-sc", + "#us-east-tn", + "#us-fl", + "#us-ga", + "#us-ga-atl", + "#us-ga-ne", + "#us-ga-northeast", + "#us-ga-nw", + "#us-gc", + "#us-gpt", + "#us-la", + "#us-la-lc-mm", + "#us-la-msy", + "#us-la-msy-mm", + "#us-lwk", + "#us-mci", + "#us-mhk", + "#us-mo", + "#us-ms", + "#us-ms-gpt-mm", + "#us-msy-jan-mm", + "#us-nc", + "#us-ny", + "#us-sc", + "#us-se", + "#us-south", + "#us-southeast", + "#us-tn", + "#us-tn-bna", + "#us-tn-cha", + "#us-tn-east", + "#us-va", + "#us-wi", + "#usa", + "#usregion", + "#ut", + "#utc", + "#va", + "#vi", + "#vis", + "#vpz", + "#vt", + "#w-wa", + "#wa", + "#wb", + "#wct", + "#wd-n-ut", + "#wd-ut", + "#west", + "#westseattle", + "#wf", + "#wi", + "#wnc", + "#wtx", + "#wv", + "#wva", + "#wy", + "#xcm", + "#xna", + "#xph", + "#yakima-co", + "#ybl", + "#ycd", + "#yeg", + "#ygk", + "#yhm", + "#yhu", + "#yhz", + "#yka", + "#ykf", + "#ykm", + "#ylk", + "#ylw", + "#yml", + "#yow", + "#ypa", + "#yqa", + "#yqb", + "#yqf", + "#yql", + "#yqq", + "#yqt", + "#yqy", + "#yrq", + "#yse", + "#ysj", + "#yt", + "#ytf", + "#ytr", + "#yul", + "#yve", + "#yvr", + "#ywg", + "#yws", + "#yxu", + "#yxx", + "#yyb", + "#yyc", + "#yyj", + "#yyy", + "#yyz", + "#zbm", + "#zmonkey", + "#zph" + ], + "regions": [ + "abe", + "abi", + "abq", + "acy", + "ags", + "aiz", + "alb", + "ama", + "aoh", + "apf", + "atl", + "atw", + "aus", + "avl", + "axn", + "azo", + "bdl", + "bfd", + "bfl", + "bgm", + "bgr", + "bhm", + "bli", + "bna", + "boi", + "bos", + "buf", + "bwi", + "cae", + "cez", + "cgi", + "cha", + "chs", + "cle", + "clt", + "cmh", + "cmi", + "coa", + "con", + "cos", + "crw", + "cvg", + "cvo", + "cwa", + "dab", + "day", + "dec", + "den", + "det", + "dfw", + "dvn", + "eat", + "eau", + "ege", + "elm", + "elp", + "eug", + "evv", + "ewn", + "far", + "fat", + "fca", + "flo", + "fnl", + "fnt", + "fpr", + "fwa", + "gdw", + "geg", + "gjt", + "gpt", + "gpz", + "grr", + "gso", + "gsp", + "gtr", + "guf", + "hlg", + "hou", + "hts", + "hyr", + "iad", + "ida", + "ilm", + "ind", + "jan", + "jax", + "kls", + "las", + "lax", + "lbb", + "lch", + "lcq", + "lex", + "lft", + "lgu", + "lmt", + "lns", + "loz", + "lru", + "lse", + "mac", + "maf", + "mbs", + "mce", + "mci", + "mdt", + "mem", + "mfe", + "mfr", + "mgm", + "mke", + "mlb", + "mlu", + "mmh", + "mmv", + "mot", + "msn", + "msp", + "msv", + "msy", + "mtj", + "ntu", + "nyc", + "oak", + "ocf", + "ogg", + "okc", + "oma", + "onp", + "ord", + "orl", + "oxr", + "pae", + "pah", + "pdx", + "pgd", + "pgv", + "phx", + "pia", + "pih", + "pit", + "pkb", + "pns", + "ppm", + "prb", + "prc", + "psc", + "psf", + "psk", + "psp", + "pub", + "puw", + "pvd", + "rdd", + "rdm", + "rdu", + "rfd", + "ric", + "rkd", + "rks", + "rno", + "roa", + "roc", + "rst", + "rsw", + "rut", + "san", + "sat", + "sav", + "sba", + "sbn", + "sbp", + "sby", + "sce", + "sdf", + "sea", + "sfo", + "sgf", + "sgu", + "sjc", + "sjt", + "slc", + "sle", + "smf", + "smn", + "spi", + "srq", + "stc", + "stl", + "svc", + "syr", + "tlh", + "tpa", + "tri", + "ttd", + "tul", + "twf", + "txk", + "tys", + "vis", + "vpz", + "xcm", + "xna", + "xph", + "ybl", + "ycd", + "yeg", + "ygk", + "yhm", + "yhu", + "yhz", + "yka", + "ykf", + "ylk", + "yml", + "yow", + "ypa", + "yqa", + "yqb", + "yqf", + "yql", + "yqq", + "yqt", + "yqy", + "yrq", + "yse", + "ysj", + "ytf", + "ytr", + "yul", + "yve", + "yvr", + "ywg", + "yws", + "yxu", + "yxx", + "yyb", + "yyc", + "yyj", + "yyy", + "yyz", + "zbm", + "zph" + ], + "groups": [ + "baus", + "chgo", + "chih", + "co", + "ct", + "ei", + "fl", + "golm", + "hb", + "maine", + "ncmc", + "onqc", + "pnw", + "socal", + "swbc", + "swoh" + ], + "province_territory_codes": [ + "ab", + "bc", + "mb", + "nb", + "nl", + "ns", + "nt", + "nu", + "on", + "pe", + "qc", + "sk", + "yt" + ], + "us_states": [ + "al", + "ak", + "az", + "ar", + "ca", + "co", + "ct", + "de", + "fl", + "ga", + "hi", + "id", + "il", + "in", + "ia", + "ks", + "ky", + "la", + "me", + "md", + "ma", + "mi", + "mn", + "ms", + "mo", + "mt", + "ne", + "nv", + "nh", + "nj", + "nm", + "ny", + "nc", + "nd", + "oh", + "ok", + "or", + "pa", + "ri", + "sc", + "sd", + "tn", + "tx", + "ut", + "vt", + "va", + "wa", + "wv", + "wi", + "wy" + ], + "us_district_territory_codes": [ + "dc", + "as", + "gu", + "mp", + "pr", + "vi" + ], + "explicit_extras": [ + "ca", + "can", + "us", + "usa", + "na" + ], + "published_scope_names": [ + "#ATW", + "#CHA", + "#CLT", + "#CVG", + "#CWA", + "#EAU", + "#EWN", + "#GDW", + "#GRR", + "#GSO", + "#ILM-MM", + "#LSE", + "#MHK", + "#MKE", + "#MSN", + "#MSP", + "#ORD", + "#RFD", + "#US-EAST-1", + "#US-EAST-AL", + "#US-EAST-FL", + "#US-EAST-GA", + "#US-EAST-NC", + "#US-EAST-SC", + "#US-EAST-TN", + "#abi", + "#ast", + "#atl", + "#atw", + "#az", + "#azo", + "#bay-area", + "#bc", + "#bdl", + "#bend", + "#bli", + "#bna", + "#bob", + "#boi", + "#bos", + "#brk", + "#broom", + "#bvs", + "#bw", + "#c-or", + "#c-ut", + "#c-wa", + "#can", + "#capitolhill", + "#cda", + "#ced", + "#cha", + "#chgo", + "#cls", + "#coast-or", + "#ct", + "#ct-ffc", + "#ct-rv", + "#cv", + "#cvo", + "#cwa", + "#default", + "#den", + "#dij", + "#e-id", + "#e-idregion", + "#east", + "#eat", + "#eau", + "#eid", + "#erc", + "#ercregion", + "#eug", + "#fl", + "#frd", + "#ga", + "#gc", + "#gc-la", + "#gc-la-msy", + "#gc-la-msy-mm", + "#gc-la-msy-sja", + "#gdw", + "#geg", + "#grh", + "#grr", + "#gso", + "#gtp", + "#hg", + "#hou", + "#howltest", + "#hunting-group", + "#hv", + "#id", + "#ida", + "#idaho", + "#imw", + "#inw", + "#jac", + "#keh", + "#kit", + "#kls", + "#las", + "#lgu", + "#lmt", + "#local", + "#lse", + "#lwc", + "#ma", + "#mad", + "#manitowoc", + "#mci", + "#me", + "#mfr", + "#mhk", + "#mke", + "#mmap", + "#mmv", + "#mo", + "#msn", + "#msp", + "#mt", + "#n-ut", + "#nc", + "#nh", + "#northeast", + "#nv", + "#nwtx", + "#nyc", + "#ogd", + "#olm", + "#oly", + "#on", + "#onp", + "#onqc", + "#or", + "#ord", + "#oth", + "#otk", + "#ottawa", + "#pae", + "#pdx", + "#pgv", + "#pia", + "#pid", + "#pih", + "#pnw", + "#psc", + "#pv", + "#rdm", + "#rdu", + "#rfd", + "#ririe", + "#rks", + "#rld", + "#rxe", + "#s-id", + "#s-ut", + "#save", + "#sbn", + "#sc", + "#se-", + "#se-wa", + "#sea", + "#seattle", + "#setx", + "#sgu", + "#sjt", + "#slc", + "#sle", + "#smn", + "#snoco", + "#southeast", + "#southseattle", + "#stg", + "#sun", + "#sun-prairie", + "#sw-id", + "#sw-wa", + "#swbc", + "#tdo", + "#test", + "#test-1e", + "#test-b5", + "#test-cc", + "#texas", + "#tn", + "#twf", + "#twn", + "#tworivers", + "#united-states", + "#us", + "#us-al", + "#us-ca", + "#us-cm", + "#us-cm-mm", + "#us-east", + "#us-east-1", + "#us-east-al", + "#us-east-fl", + "#us-east-ga", + "#us-east-nc", + "#us-east-sc", + "#us-east-tn", + "#us-fl", + "#us-ga", + "#us-ga-atl", + "#us-ga-ne", + "#us-ga-northeast", + "#us-ga-nw", + "#us-gc", + "#us-gpt", + "#us-la", + "#us-la-lc-mm", + "#us-la-msy", + "#us-la-msy-mm", + "#us-lwk", + "#us-mci", + "#us-mhk", + "#us-mo", + "#us-ms", + "#us-ms-gpt-mm", + "#us-msy-jan-mm", + "#us-nc", + "#us-ny", + "#us-sc", + "#us-se", + "#us-south", + "#us-southeast", + "#us-tn", + "#us-tn-bna", + "#us-tn-cha", + "#us-tn-east", + "#us-va", + "#us-wi", + "#usregion", + "#ut", + "#utc", + "#va", + "#w-wa", + "#wa", + "#wb", + "#wct", + "#wd-n-ut", + "#wd-ut", + "#west", + "#westseattle", + "#wf", + "#wi", + "#wnc", + "#wtx", + "#wv", + "#wva", + "#yakima-co", + "#ycd", + "#yka", + "#ykm", + "#ylw", + "#yow", + "#yqb", + "#yul", + "#zmonkey" + ], + "published_sources_checked": 237, + "countries": [ + "CA", + "US" + ], + "source_urls": [ + "https://canadaverse.org/beacon-dev/api/v1/iatas", + "https://meshmapper.net/get_zones.php?country=CA", + "https://meshmapper.net/get_zones.php?country=US" + ], + "update_policy": "Explicit snapshot audit; existing YOW automatic importer remains enabled", + "checked_at": "2026-09-28", + "note": "Candidate names only; exact case matters. Unknown and ambiguous transport codes remain unresolved." +} From a3807ffc6f48d432256d764616e89893213d94c5 Mon Sep 17 00:00:00 2001 From: n30nex Date: Mon, 28 Sep 2026 17:25:24 -0400 Subject: [PATCH 32/69] docs: fix encoding and clarify optional stack workflow --- CONTRIBUTOR_WORKFLOW.md | 29 +++++++++++++++++++++++++---- RELEASE-CHECKLIST.md | 14 +++++++++----- ROADMAP.md | 12 ++++++++++-- 3 files changed, 44 insertions(+), 11 deletions(-) diff --git a/CONTRIBUTOR_WORKFLOW.md b/CONTRIBUTOR_WORKFLOW.md index 045e258..fef4df4 100644 --- a/CONTRIBUTOR_WORKFLOW.md +++ b/CONTRIBUTOR_WORKFLOW.md @@ -1,6 +1,20 @@ # Contributor workflow -Use this workflow to keep small Beacon changes reviewable while reducing manual branch maintenance. The executable helper is [tools/beacon_stack.py](tools/beacon_stack.py); a CI workflow is included for its [offline regression tests](tools/test_beacon_stack.py). +Use this workflow to keep small Beacon changes reviewable while reducing manual branch maintenance. The optional contributor helper is [tools/beacon_stack.py](tools/beacon_stack.py); a CI workflow is included for its [offline regression tests](tools/test_beacon_stack.py). Beacon does not run this script during ingestion or at runtime. Maintainers can review and merge through GitHub without installing it. + +## Merge commits and when the helper is useful + +The current stacks include commits from still-unmerged parent PRs. Squash merging replaces a parent's commits with a new commit; GitHub's rebase-and-merge also creates new SHAs. Children still contain the old parents. The helper records each feature's exact delta so it can repair that history once for the whole queue. It also composes the preview with independent fixes and verifies source and CI identities. + +For dependent PRs, **merge commits are the recommended way to retain ancestry**. Merge the parent first, then inspect the child's updated diff and checks. The parent's original commits are now in `dev`, so that history alone no longer requires rewriting the child. Squashing an independent PR without pending descendants remains an option. GitHub documents these [merge methods and the long-running-branch tradeoff](https://docs.github.com/en/pull-requests/reference/pull-request-merges). + +This reduces routine stack maintenance; it does not remove source conflicts, API dependencies or required validation. A rule requiring branches to be current can still require an update, and a linear-history rule would prohibit merge commits. Enabling the repository option does not repair ancestors that were already squashed/rebased. Merge in dependency order and inspect each remaining PR; do not merge a child early merely because it includes its parent. + +At the 28 September check, both application repos allowed squash/rebase merging and disabled merge commits. Changing that repository setting is a maintainer decision. The visible `dev` rulesets did not list linear-history or up-to-date status-check requirements, but the legacy branch-protection endpoints returned 404; that response is not proof that no other policy applies. + +The helper remains useful for the existing queue's integration with new `dev` changes, squash/rebase recovery, conflict isolation, exact combined-preview validation and leased fork updates. It is not an extra GitHub merge gate. Its current `Check`/`Publish` modes deliberately require a refreshed `dev` base; enabling merge commits does not silently change that implementation or make an old preview current. Use ordinary GitHub merging when its checks and dependencies permit it, and use the helper when preparing a current composed candidate. There is no need to run Sync merely to let a maintainer click Merge. + +The normal long-term path is a short queue: finish the current review sequence, then start each independent change from fresh `dev`. Keep only real dependencies stacked. Retain the exact built source and existing rollback; a documentation or history-only change with the same source tree needs no application rebuild. ## Working loop @@ -9,8 +23,8 @@ Use this workflow to keep small Beacon changes reviewable while reducing manual 3. Follow each repository's contribution rules. Prefer existing components and feature-owned files; keep shared startup/router/navigation changes in a declared order. 4. Build and test the change, open its PR against `dev`, then record its exact parent, head, fork branch and worktree in the manifest. Link the parent-to-head comparison in the PR body. 5. Keep current Beacon contribution PRs out of draft as requested by the contributor, with dependencies visible. Request MrAlders0n's review; if account permissions prevent formal assignment, use an explicit review-request comment instead. -6. Keep cross-repository rollout order explicit: a new web page waits until its server endpoint is merged **and deployed**. Ready for review does not imply ready to merge. -7. After merges, run Sync and Check. Rebuild/redeploy the preview only if the composed source changes, preserving the actual running revision and corresponding-source offer. +6. Keep cross-repository rollout order explicit: deploy a required server API before its web consumer. Development previews may compose declared review candidates; maintainers decide acceptance and production release. Ready for review does not imply ready to merge. +7. After merges, inspect status and the remaining dependencies. For a refreshed composed preview or history repair, run Sync and Check once for the queue. GitHub review/merge does not require this script. Rebuild/redeploy only if the composed source changes, preserving the actual running revision and corresponding-source offer. A source conflict still needs review. The helper automates routine history movement; it does not promise that overlapping edits can never conflict, merge upstream PRs, deploy services, or create scheduled jobs. @@ -18,7 +32,10 @@ A source conflict still needs review. The helper automates routine history movem The 27 September refresh moved #166 independently and #167 -> #169 together onto accepted server #170. Route evidence #172 follows #169 and MeshMapper scope import #174 follows #172; its web consumer #85 follows #83, and navigation #87 follows #85. The Pi composition includes every current candidate. Refresh/Publish/Check handles this ancestry once; source conflicts still require review. A history-only change with an identical tree still needs no Pi rebuild. -Current preview is server `eb99f752` / web `98f820d2`, with native PostgreSQL and exact-head CI/race/security checks. Every existing candidate is retained; only YOW scope import is enabled. Rollback restores original schema041 database `beacon_pre042_20260927` and its matching server/configuration/source. The frontend is unchanged. Follow the [roadmap](ROADMAP.md) for issues, evidence and the next channel-scope slice. +The public preview manifest verified on 28 September is server `7c9599b1` / web `dfeb2777`, including all current review candidates, the Public key and 504 scope candidates. Its validation and layered recovery are recorded in the [roadmap](ROADMAP.md); use the exact deployed records rather than older examples in this document. + +Upstream subsequently accepted ingest buffering #179 and route-reconfirmation batching #180; current server `dev` is `db30c9b5`, and web `dev` is `17f48fb9`. These changes are not in that preview. Their integration and native ingest/route-lock regression checks take priority over the next feature. The helper still reports both queues as needing refresh. No runtime, repository merge setting or helper algorithm was changed by this documentation correction. + ## Setup @@ -103,3 +120,7 @@ python -m unittest discover -s tools -p test_beacon_stack.py -v ``` Tests cover squash/drop-parent behavior, a fresh phase after all merges, cache reuse, environment changes, independent overlay CI/state/identity checks, missing or pending checks, skipped-job policy, publication races, dirty/default-branch rejection, source conflicts and upstream movement during validation. They use disposable local Git repositories and no GitHub or Pi credentials. + +## Document encoding + +Markdown is UTF-8. Read and write it with an explicit UTF-8 encoding in scripts, especially when moving between Windows tools. Check both the diff and rendered text before publication. Keep numeric ranges as en dashes and dependency arrows as arrows; do not round-trip the document through a legacy Windows code page. diff --git a/RELEASE-CHECKLIST.md b/RELEASE-CHECKLIST.md index 4a43607..b420ce7 100644 --- a/RELEASE-CHECKLIST.md +++ b/RELEASE-CHECKLIST.md @@ -1,6 +1,10 @@ # Server/web consolidation release -Status: observer-first development preview and review handoff, 27 September 2026. Stable tags and production cutover remain owner-managed. This is not a complete CoreScope parity claim. +Status: development preview and review handoff, 28 September 2026. Stable tags and production cutover remain owner-managed. This is not a complete CoreScope parity claim. + +## Current integration gate — 28 September + +The preview remains server `7c9599b1` / web `dfeb2777`. Before the next release candidate, integrate upstream ingest buffering #179 and route-reconfirmation batching #180 (`dev` now `db30c9b5`), refresh the dependent queue once, and repeat native burst/route-lock contention checks on the resulting combination. Existing fixture results and the older checkpoints below do not establish lossless production ingestion. Consult the [current roadmap](ROADMAP.md) and [merge workflow](CONTRIBUTOR_WORKFLOW.md) for the queue and recovery boundaries. ## Packet reception investigation — 27 September @@ -8,13 +12,13 @@ Status: observer-first development preview and review handoff, 27 September 2026 Map projection omits ambiguous/unlocated identities and breaks lines at gaps. Live animations across uncertain chains are suppressed, so fewer speculative lines appear. Unavailable selected paths no longer silently show All paths. A shared-path loading race is fixed by checking the requested packet hash. Packet labels use the existing Noto Sans stack; external basemap emoji-glyph/sprite fallback warnings can still occur. -The Pi now runs web `1d5d65e` with unchanged server `88c2c10c`. Native build/lint and all **906 tests** pass; focused Windows checks and desktop/390px phone/English/French/keyboard/Back/shared-link checks pass. Public assets and source match, both MQTT feeds are connected, and the frontend publication restarted no services. Earlier review candidates remain included. The [changelog/source](https://canadaverse.org/beacon-dev/source.html) identifies the running build. Maintainers still own merges, stable releases and production cutover. +At the packet-investigation checkpoint, the Pi ran web `1d5d65e` with unchanged server `88c2c10c`. Native build/lint and all **906 tests** pass; focused Windows checks and desktop/390px phone/English/French/keyboard/Back/shared-link checks pass. Public assets and source match, both MQTT feeds are connected, and the frontend publication restarted no services. Earlier review candidates remain included. The [changelog/source](https://canadaverse.org/beacon-dev/source.html) identifies the running build. Maintainers still own merges, stable releases and production cutover. -**Next:** review feedback/issues first, then define a bounded known-route-to-retained-packet/report API with exact-byte versus possible-identity semantics and pagination. Non-packet overlay return navigation remains a separate follow-up. MeshMapper scope import remains a draft pending an agreed public endpoint/schema. Broader server #60/#72/#99/#116 and web #12 remain open; this is a first connected-investigation slice, not full parity. +**Historical follow-up:** route evidence, observer return navigation and MeshMapper scope import were subsequently delivered as review candidates; see the current roadmap. Broader server #60/#72/#99/#116 and web #12 remain open; this is a first connected-investigation slice, not full parity. ## Current observer release -The observer-first release is implemented in four focused review candidates: [server #169](https://github.com/MeshCore-Beacon/beacon-server/pull/169) (`60ed339c`, closes #168), [web #79](https://github.com/MeshCore-Beacon/beacon-web/pull/79) (`3a0eb4c8`, closes #76), [web #80](https://github.com/MeshCore-Beacon/beacon-web/pull/80) (`b3a6b088`, closes #77) and [web #81](https://github.com/MeshCore-Beacon/beacon-web/pull/81) (`42ae09b7`, closes #78). All are out of draft. Server #169 follows #167; web order is #75 → #79 → #80 → #81. Independent server #166 remains in the preview composition. Maintainers control acceptance and release; these issues remain open until their changes are accepted. +The observer-first release is implemented in four focused review candidates: [server #169](https://github.com/MeshCore-Beacon/beacon-server/pull/169) (`91b21995`, closes #168), [web #79](https://github.com/MeshCore-Beacon/beacon-web/pull/79) (`3a0eb4c8`, closes #76), [web #80](https://github.com/MeshCore-Beacon/beacon-web/pull/80) (`b3a6b088`, closes #77) and [web #81](https://github.com/MeshCore-Beacon/beacon-web/pull/81) (`42ae09b7`, closes #78). All are out of draft. Server #169 follows #167; web order is #75 → #79 → #80 → #81. Independent server #166 remains in the preview composition. Maintainers control acceptance and release; these issues remain open until their changes are accepted. At the observer-release checkpoint the Pi ran composed server `88c2c10c830034cee70a46fca717af518803a544` and web `42ae09b7c6be50cd0f617bad8aea35825be9f12e`, with the [updated changelog and exact source](https://canadaverse.org/beacon-dev/source.html). Raw packets remain 72 hours, archived hourly analytics 30 days, telemetry 720 hours. All four candidates passed their native Pi suites; the final web build passes 895 tests. Server tests use actual PostgreSQL. Windows server/full destination/dashboard validation and 48 focused final comparison/API tests also pass. Desktop, 390-pixel phone, English/French, keyboard, legacy/shared links and Back/search/sort/scroll were checked. Physical iPhone Safari and production-volume capacity remain separate gates. @@ -91,7 +95,7 @@ Current dev is server `91b4b457` / web `54b5093a`, with passing CI/image builds - [x] September 20 unmodified Pi stability sample: 600 seconds / 41 samples, both feeds connected, 2,169 retained observations and no MQTT disconnect/deadline or HTTP 5xx. App/PostgreSQL CPU averaged 2.14%/3.96% of one core. This is a bounded health sample, not callback timing, #116 root-cause proof or a production-volume gate. [Result and limits](https://github.com/MeshCore-Beacon/beacon-server/issues/116#issuecomment-5753626821). - [x] Native Pi build/test of server `6be0f762` and web `42ba5fc`, including real PostgreSQL-to-HTTP checks and migration retry/concurrent refresh; 786 web tests pass. -- [x] One-million-row request/initial-population/refresh/storage measurements; request plans read only the new views. Signal: 1.8–77.5 ms reads, 14.4 s initial population, 16.8 s refresh, 6.5 MB. Paths: 3.5–141.9 ms reads, 8.4 s population, 6.2 s refresh, 11.3 MB. +- [x] One-million-row request/initial-population/refresh/storage measurements; request plans read only the new views. Signal: 1.8–77.5 ms reads, 14.4 s initial population, 16.8 s refresh, 6.5 MB. Paths: 3.5–141.9 ms reads, 8.4 s population, 6.2 s refresh, 11.3 MB. - [ ] Measure the full operator workload and sustained refresh/ingest load before a production parity claim. The million-row fixture does not establish that limit. - [x] Backup client mismatch and unsupported-DSN cases leave the public API available; valid client export/restore used disposable data only and restored all 35 source migrations. Public preview admin/backup remains disabled. - [x] Real preview analytics reconcile with SQL for the materialized window. Browser charts, complete-hour text, small screens and error/empty/retry states are verified. Both MQTT feeds advance and the public browser reports LIVE. diff --git a/ROADMAP.md b/ROADMAP.md index 69fbf12..f01f8c1 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -6,7 +6,15 @@ Refresh GitHub issues, PR feedback and branch state before starting a phase. Thi Release-check correction, 21 September UTC: the workflow now includes independent preview PRs in Status and Check, applies the same CI/head/fork/target requirements to them, and rechecks the prepared independent inputs before publication. This covers packet summaries #161 and map correction #61 without adding them to the ordered stacks. The backup CLI #160 retains its separate check. Twenty-three offline regressions cover these gates and the existing no-rebase/cache behavior. See [the contributor workflow](CONTRIBUTOR_WORKFLOW.md). -Latest upstream is server `876d9970` (#177/#178) and web `17f48fb9` (#91). The tested preview retains its exact integrated bases; both review stacks require one helper-managed refresh before the next publication. Do not rebase children independently or relabel existing binaries. +Latest upstream is server `db30c9b5` (#177–#180) and web `17f48fb9` (#91). The tested preview retains its exact integrated bases; both review stacks require one helper-managed refresh before the next publication. Do not rebase children independently or relabel existing binaries. + +## Maintainer feedback and ingest priority — 28 September + +The reported garbled ranges and dependency arrows were encoding mistakes in the shared docs. They are corrected as UTF-8 without changing the measurements. The [contributor workflow](CONTRIBUTOR_WORKFLOW.md) now explains why squash/rebase history required the helper and recommends merge commits for dependent PRs. The helper is a contributor integration tool, not part of Beacon ingestion or a maintainer merge prerequisite. Both application repos still have merge commits disabled; maintainers control that setting. No helper behavior or repository settings changed in this documentation update. + +Current accepted server `dev` is `db30c9b573357990c41166292e7cf42785c92cc4`, including [#179](https://github.com/MeshCore-Beacon/beacon-server/pull/179) and [#180](https://github.com/MeshCore-Beacon/beacon-server/pull/180). #179 uses bounded workers, preserves each observer’s message order and reduces database work. #180 limits reconfirmation transactions to 1,000 routes, uses a five-second batch timeout and skips busy rows after the maintainer identified ingest blocking. Their published replay/load results are maintainer evidence for those fixtures, not an independently verified production losslessness claim or an explanation of the older CoreScope counter difference. + +**Next:** integrate the latest ingest/route changes with the pending review stack once, resolve real source conflicts deliberately, and validate the resulting native Pi build under burst and route-lock contention before resuming feature work. The public preview remains server `7c9599b1` / web `dfeb2777`; it has not yet received #179/#180. Both queues still need a current-base refresh. Owners retain merges and production release. ## Canada/US scope coverage and packet layout — 28 September @@ -166,7 +174,7 @@ Current broader issues remain server #60 (admin), #72 (backup/import), #99 (pack The path page counts stored receptions, not unique devices. Flood paths accumulate entries, while direct routes carry remaining entries. Observed widths do not establish device capability or collision rates. Signal readings describe reception at the reporting observer rather than a complete end-to-end link. -The September 20 review correction moves both new aggregate APIs onto materialized hourly snapshots, preserving reception/region semantics and normalizing polling windows to UTC hours. In a rolled-back million-row Pi fixture, Signal request queries took 1.8–77.5 ms and Paths 3.5–141.9 ms across custom/generic plans. Initial population took 14.4/8.4 seconds, refresh 16.8/6.2 seconds, and view/index storage was 6.5/11.3 MB respectively. These measurements cover the fixture, not the full production workload. See the [release consolidation checklist](RELEASE-CHECKLIST.md) for remaining gates. +The September 20 review correction moves both new aggregate APIs onto materialized hourly snapshots, preserving reception/region semantics and normalizing polling windows to UTC hours. In a rolled-back million-row Pi fixture, Signal request queries took 1.8–77.5 ms and Paths 3.5–141.9 ms across custom/generic plans. Initial population took 14.4/8.4 seconds, refresh 16.8/6.2 seconds, and view/index storage was 6.5/11.3 MB respectively. These measurements cover the fixture, not the full production workload. See the [release consolidation checklist](RELEASE-CHECKLIST.md) for remaining gates. September 20 validation covered native Go/PostgreSQL/HTTP behavior and **786 web tests**, plus private backup compatibility, feature-only startup failure, TLS/password files, cancellation and schema/data/sequence restoration. That review update passed 390/1280px browser checks, with ten distinct glyphs and complete-hour text; earlier chart checks also covered 320/768px. This is historical evidence for the unchanged feature code. Current revisions and corresponding-source archives are on the preview's changelog page. From a55bfe625cb493a3fbfc41eff5f6c1e25f1b54f6 Mon Sep 17 00:00:00 2001 From: n30nex Date: Mon, 28 Sep 2026 17:29:51 -0400 Subject: [PATCH 33/69] chore: refresh documentation review head From dbc1c397fc3786782c456e096e8f7cb80a458db9 Mon Sep 17 00:00:00 2001 From: n30nex Date: Mon, 28 Sep 2026 19:02:13 -0400 Subject: [PATCH 34/69] docs: record validated ingest integration and next timestamp fix --- CONTRIBUTOR_WORKFLOW.md | 7 ++- RELEASE-CHECKLIST.md | 8 +++- ROADMAP.md | 41 +++++++++------- .../ingest-integration-20260928.md | 47 +++++++++++++++++++ 4 files changed, 80 insertions(+), 23 deletions(-) create mode 100644 app_documentation/ingest-integration-20260928.md diff --git a/CONTRIBUTOR_WORKFLOW.md b/CONTRIBUTOR_WORKFLOW.md index fef4df4..1c4e469 100644 --- a/CONTRIBUTOR_WORKFLOW.md +++ b/CONTRIBUTOR_WORKFLOW.md @@ -30,12 +30,11 @@ A source conflict still needs review. The helper automates routine history movem ## Current integration example -The 27 September refresh moved #166 independently and #167 -> #169 together onto accepted server #170. Route evidence #172 follows #169 and MeshMapper scope import #174 follows #172; its web consumer #85 follows #83, and navigation #87 follows #85. The Pi composition includes every current candidate. Refresh/Publish/Check handles this ancestry once; source conflicts still require review. A history-only change with an identical tree still needs no Pi rebuild. +The 28 September coordinated refresh integrates server dev `db30c9b5` (#177–#180) and web dev `17f48fb9` (#91). Server #166 stays independent; #167 -> #169 -> #172 -> #174 -> #176 is the ordered server chain. Web #75 -> #79 -> #80 -> #81 -> #83 -> #85 -> #87 -> #89 -> #92 remains one declared chain. Only #166 had a source conflict; it was resolved with advert/repeat regressions before publication. Both full-stack Check commands pass for all fifteen exact published heads. -The public preview manifest verified on 28 September is server `7c9599b1` / web `dfeb2777`, including all current review candidates, the Public key and 504 scope candidates. Its validation and layered recovery are recorded in the [roadmap](ROADMAP.md); use the exact deployed records rather than older examples in this document. - -Upstream subsequently accepted ingest buffering #179 and route-reconfirmation batching #180; current server `dev` is `db30c9b5`, and web `dev` is `17f48fb9`. These changes are not in that preview. Their integration and native ingest/route-lock regression checks take priority over the next feature. The helper still reports both queues as needing refresh. No runtime, repository merge setting or helper algorithm was changed by this documentation correction. +The Pi runs composed server `2ed2e031` / web `6b688495`, retaining every current feature. Native database/queue/route-lock checks and all 935 web tests pass. All 18 rebuilt web assets are byte-identical to the prior geometry-tested build; the corresponding source offer identifies the actual refreshed source. Same-schema rollback preserves new rows. See [validation and exact PR heads](app_documentation/ingest-integration-20260928.md). +Merge commits remain disabled in both application repos, and the contributor has READ access. A maintainer must enable that option. This integration changes no merge policy or helper algorithm; it uses the existing isolated conflict stop, cached validation, leased publication and exact-head checks. Native tests and public/runtime checks supplement CI when combined behavior changes. ## Setup diff --git a/RELEASE-CHECKLIST.md b/RELEASE-CHECKLIST.md index b420ce7..1164afd 100644 --- a/RELEASE-CHECKLIST.md +++ b/RELEASE-CHECKLIST.md @@ -4,7 +4,11 @@ Status: development preview and review handoff, 28 September 2026. Stable tags a ## Current integration gate — 28 September -The preview remains server `7c9599b1` / web `dfeb2777`. Before the next release candidate, integrate upstream ingest buffering #179 and route-reconfirmation batching #180 (`dev` now `db30c9b5`), refresh the dependent queue once, and repeat native burst/route-lock contention checks on the resulting combination. Existing fixture results and the older checkpoints below do not establish lossless production ingestion. Consult the [current roadmap](ROADMAP.md) and [merge workflow](CONTRIBUTOR_WORKFLOW.md) for the queue and recovery boundaries. +The Pi preview now runs server `2ed2e031` / web `6b688495`, integrating upstream server #177–#180 and web #91 with all fifteen pending candidates. Both queues pass current-base Check and actual published-head CI (web CodeQL skipped). Full native server/PostgreSQL and 935 web tests pass. The isolated 3,200-input/504-scope replay preserves expected rows and events under route-lock contention with zero drops; both actual inputs advanced during a 63-second observation. This does not establish universal lossless production ingestion or explain the historical counter difference. + +Fresh source/18-asset verification and desktop/French-phone smoke checks pass. Only the Beacon app restarted; other services and schema042/configuration stayed intact. A fresh dump restored and was verified off-Pi; current and rollback binary/frontend/source remain. See [exact heads, evidence and recovery](app_documentation/ingest-integration-20260928.md). + +Remaining before further feature work: review feedback, then [server #181](https://github.com/MeshCore-Beacon/beacon-server/issues/181) timestamp parsing. Input timestamp/region warnings remain. Physical iPhone Safari, production-volume capacity, broad partial issues and maintainer release acceptance remain distinct gates. Enabling merge commits requires a maintainer; the contributor's account is READ. ## Packet reception investigation — 27 September @@ -16,7 +20,7 @@ At the packet-investigation checkpoint, the Pi ran web `1d5d65e` with unchanged **Historical follow-up:** route evidence, observer return navigation and MeshMapper scope import were subsequently delivered as review candidates; see the current roadmap. Broader server #60/#72/#99/#116 and web #12 remain open; this is a first connected-investigation slice, not full parity. -## Current observer release +## Observer release checkpoint — 27 September The observer-first release is implemented in four focused review candidates: [server #169](https://github.com/MeshCore-Beacon/beacon-server/pull/169) (`91b21995`, closes #168), [web #79](https://github.com/MeshCore-Beacon/beacon-web/pull/79) (`3a0eb4c8`, closes #76), [web #80](https://github.com/MeshCore-Beacon/beacon-web/pull/80) (`b3a6b088`, closes #77) and [web #81](https://github.com/MeshCore-Beacon/beacon-web/pull/81) (`42ae09b7`, closes #78). All are out of draft. Server #169 follows #167; web order is #75 → #79 → #80 → #81. Independent server #166 remains in the preview composition. Maintainers control acceptance and release; these issues remain open until their changes are accepted. diff --git a/ROADMAP.md b/ROADMAP.md index f01f8c1..41de3cf 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -6,17 +6,23 @@ Refresh GitHub issues, PR feedback and branch state before starting a phase. Thi Release-check correction, 21 September UTC: the workflow now includes independent preview PRs in Status and Check, applies the same CI/head/fork/target requirements to them, and rechecks the prepared independent inputs before publication. This covers packet summaries #161 and map correction #61 without adding them to the ordered stacks. The backup CLI #160 retains its separate check. Twenty-three offline regressions cover these gates and the existing no-rebase/cache behavior. See [the contributor workflow](CONTRIBUTOR_WORKFLOW.md). -Latest upstream is server `db30c9b5` (#177–#180) and web `17f48fb9` (#91). The tested preview retains its exact integrated bases; both review stacks require one helper-managed refresh before the next publication. Do not rebase children independently or relabel existing binaries. +Latest integrated upstream is server `db30c9b5` (#177–#180) and web `17f48fb9` (#91). One coordinated Refresh/Publish/Check pass is complete. The Pi now runs composed server `2ed2e031` / web `6b688495`, including all fifteen review candidates; use the current heads below rather than historical checkpoint SHAs. Recheck upstream before new work. -## Maintainer feedback and ingest priority — 28 September +## Ingest integration delivered — 28 September -The reported garbled ranges and dependency arrows were encoding mistakes in the shared docs. They are corrected as UTF-8 without changing the measurements. The [contributor workflow](CONTRIBUTOR_WORKFLOW.md) now explains why squash/rebase history required the helper and recommends merge commits for dependent PRs. The helper is a contributor integration tool, not part of Beacon ingestion or a maintainer merge prerequisite. Both application repos still have merge commits disabled; maintainers control that setting. No helper behavior or repository settings changed in this documentation update. +The upstream ingest queue and bounded route-reconfirmation changes are integrated with all pending features. Independent server #166 was updated to preserve advert-name freshness and optional repeat-path delivery together. Suppressed duplicates perform no live-only endpoint work; repeats do not overwrite current names with older adverts. The rest of the queue refreshed without source conflicts. -Current accepted server `dev` is `db30c9b573357990c41166292e7cf42785c92cc4`, including [#179](https://github.com/MeshCore-Beacon/beacon-server/pull/179) and [#180](https://github.com/MeshCore-Beacon/beacon-server/pull/180). #179 uses bounded workers, preserves each observer’s message order and reduces database work. #180 limits reconfirmation transactions to 1,000 routes, uses a five-second batch timeout and skips busy rows after the maintainer identified ingest blocking. Their published replay/load results are maintainer evidence for those fixtures, not an independently verified production losslessness claim or an explanation of the older CoreScope counter difference. +All fifteen published application heads pass checks (web CodeQL remains skipped). The composed native Pi server/PostgreSQL suite passes. With 504 scope candidates and blocked route maintenance, an isolated 3,200-input replay preserved the expected 100 packets, 800 observations, 100 decrypted messages and 2,400 opt-in live events with zero queue drops. Native web build/lint and all 935 tests pass. All 18 public assets and both source archives match; English desktop, French phone, scoped packet rows, Public history and packet inspection pass. This is bounded fixture evidence, not proof of universal production losslessness. -**Next:** integrate the latest ingest/route changes with the pending review stack once, resolve real source conflicts deliberately, and validate the resulting native Pi build under burst and route-lock contention before resuming feature work. The public preview remains server `7c9599b1` / web `dfeb2777`; it has not yet received #179/#180. Both queues still need a current-base refresh. Owners retain merges and production release. +Both real MQTT feeds advanced during the 63-second runtime observation with no restarts or queue-overflow errors. Malformed region and timestamp warnings remain. Valid whole-second timestamps with numeric UTC offsets expose a pre-existing parser gap, now [server #181](https://github.com/MeshCore-Beacon/beacon-server/issues/181). Fix that next, after checking new reviews/issues; then resume optional MeshMapper boundaries. The older CoreScope legacy-counter gap remains unexplained. -## Canada/US scope coverage and packet layout — 28 September +Current preview: server `2ed2e03117f6c88795d446456e6d74c20c485d28` / web `6b6884951a3dac01b592dfec83f0191879c5696c`. Configuration, schema042, Public key, 504 exact-case candidates, YOW importer and 72h/30d/720h retention are unchanged. A fresh dump restored successfully and was checksum-verified off the Pi. Only Beacon restarted; the other 22 containers were unchanged. Same-schema rollback retains new data and restores server `7c9599b1` / web `dfeb2777` from `ingest-cutover-20260928T222830Z`; frontend-only recovery is `web-20260928T224458Z` paired with the new backend. Exact procedures and current PR heads are in the integration record below. + +The shared UTF-8 corrections and merge guidance remain in place. Both application repos still disable merge commits; the contributor has READ access, so a maintainer must enable that setting. The helper remains optional for maintainers and unrelated to ingestion. Owners retain upstream merges and production release. + +[Current PR heads, validation and recovery](app_documentation/ingest-integration-20260928.md). + +## Earlier Canada/US scope and packet-layout checkpoint — 28 September The preview now has **504 exact-case scope candidates**: 261 distinct names from all 237 published Canadian/US regional scope catalogues, plus lowercase IATA/group, province/state, district/territory and country fallbacks. It covers 244 currently known region codes, all 13 Canadian province/territory codes, all 50 US states, DC and five US territories, and includes `#ca`, `#can`, `#us`, `#usa` and `#na`. Counts overlap; do not add these categories together. Published mixed-case names are preserved because case changes the key. Named scopes are not geographic boundaries or evidence of repeater use. @@ -26,13 +32,13 @@ See the [candidate snapshot and provenance](app_documentation/north-american-sco [Web #92](https://github.com/MeshCore-Beacon/beacon-web/pull/92), `dfeb2777`, follows #89 and closes #90. It gives the route label and scope separate lines within a 128px track, preserving 37px rows and exact scope case. Long tags remain inside phone cards. A real browser geometry check reproduced the spill before the fix and passed afterward, including the user's `BB2F2752` row. Desktop, 768px table, 390px phone, keyboard and English/French public checks pass. All 935 tests pass on Windows and the Pi, as does exact-head CI; existing warnings remain and web CodeQL is skipped. -The Pi now serves web `dfeb277756b1a9b230d7e7e0f2d45d62713bf45b` with unchanged server `7c9599b1`, every prior candidate, and the updated [source/changelog](https://canadaverse.org/beacon-dev/source.html). All 18 assets and both source archives match. Both inputs and public LIVE are verified; frontend publication restarted no containers. Application merges/releases remain owner-controlled. New upstream server #177/#178 and web #91 require the next single helper-managed stack refresh; the running source is not relabelled as those newer heads. +At that earlier checkpoint, the Pi served web `dfeb277756b1a9b230d7e7e0f2d45d62713bf45b` with unchanged server `7c9599b1`, every prior candidate, and the updated [source/changelog](https://canadaverse.org/beacon-dev/source.html). All 18 assets and both source archives match. Both inputs and public LIVE are verified; frontend publication restarted no containers. Application merges/releases remain owner-controlled. The later coordinated refresh above integrates #177–#180 and web #91; this paragraph records the earlier validation. ## Channel scope investigation and Public channel — 27 September [Server #176](https://github.com/MeshCore-Beacon/beacon-server/pull/176) (`7fc631e8`, follows #174, closes #175) and [web #89](https://github.com/MeshCore-Beacon/beacon-web/pull/89) (`e7618fd7`, follows #87, closes #88) expose first-recorded packet scope consistently in history, catch-up and live messages. The interface adds scope filtering, packet inspection, distinct unknown/unavailable/unscoped states and English/French explanations. Duplicate broker messages and live arrivals during a history request preserve the existing page cursor and message counts. Imported catalogue names alone are not forwarding evidence. -The current Pi preview is composed server `7c9599b167bcad2b416ef03304ff27909ab3021b` / web `e7618fd7d40aff4e01f581999fdbb21d10de2e67`, with every earlier review candidate included. Native server/PostgreSQL tests, all 935 web tests, build/lint and published-head CI pass (web CodeQL is skipped). Browser checks cover live/history filtering, keyboard inspection, English/French and a 390px phone. The public page is LIVE, both MQTT inputs are connected, and all 18 assets plus both source archives match the [changelog/source offer](https://canadaverse.org/beacon-dev/source.html). +At the channel-scope checkpoint, the Pi preview was composed server `7c9599b167bcad2b416ef03304ff27909ab3021b` / web `e7618fd7d40aff4e01f581999fdbb21d10de2e67`, with every earlier review candidate included. Native server/PostgreSQL tests, all 935 web tests, build/lint and published-head CI pass (web CodeQL is skipped). Browser checks cover live/history filtering, keyboard inspection, English/French and a 390px phone. The public page is LIVE, both MQTT inputs are connected, and all 18 assets plus both source archives match the [changelog/source offer](https://canadaverse.org/beacon-dev/source.html). The standard MeshCore Public channel key is enabled at the user's request, matching the known non-hashtag hash-11 channel on dev.meshcore.ca. A restored-copy trial recovered 2,735 retained messages in 16.127 seconds; public decoded history and a new incoming message were verified. Expired packets remain unavailable. Schema042 is unchanged. Rollback retains server `eb99f752`, web `98f820d2`, configuration and source without discarding new database rows; the older schema041 recovery remains separately available. Packets stay 72h, summaries 30d and telemetry 720h. Public admin, backups and foreign detection remain disabled. @@ -72,7 +78,7 @@ Matching uses the complete saved prefix bytes, width and IATA, not confirmed phy The stack was refreshed once for accepted server #170 (`dec643a2`): optional exact WebSocket origins and opt-in observer public-key fields, not authentication. Updated server order: #167 (`02743704`) -> #169 (`91b21995`) -> #172 (`f473b165`); independent #166 (`2c5ad5fc`) remains included. Web order: #75 -> #79 -> #80 -> #81 -> #83 -> #85. All candidates remain reviewable separately; do not rebase every child independently. Deploy the server API before its web consumer. -Current preview backend is composed `99e623c56477fd9b667d5f56bfb1a0eff34ecb2b`. Its complete native Pi/PostgreSQL suite passed. Restoring a fresh private dump and adding migration 041 preserved all 31 table fingerprints. The live switch retained the original schema040 database as `beacon_pre041_20260927`; rollback directory is `route-cutover-20260927T190644Z`. Both MQTT feeds reconnected and 22 unrelated containers were unchanged. Packets remain 72h, archived summaries 30d, telemetry 720h. Public admin, backup and foreign detection remain disabled. +At the saved-route checkpoint, the preview backend was composed `99e623c56477fd9b667d5f56bfb1a0eff34ecb2b`. Its complete native Pi/PostgreSQL suite passed. Restoring a fresh private dump and adding migration 041 preserved all 31 table fingerprints. The live switch retained the original schema040 database as `beacon_pre041_20260927`; rollback directory is `route-cutover-20260927T190644Z`. Both MQTT feeds reconnected and 22 unrelated containers were unchanged. Packets remain 72h, archived summaries 30d, telemetry 720h. Public admin, backup and foreign detection remain disabled. At the route-evidence checkpoint, the Pi served web `3b3abdcc5bfa85b60c8959715736ea42c3d0bbd8`. The final native build/lint and all **915 tests** pass; exact-head CI passes (web CodeQL remains skipped). Desktop and 390px phone, English/French, copied/shared links, Back, keyboard Close/focus, exact report/node/observer and selected-map journeys were verified. All 18 public assets and both source archives match the tested artifacts. The public page is LIVE; both MQTT feeds are connected. Frontend publication restarted no containers. Web rollback retains `1d5d65e` in `web-20260927T192154Z`; the [source/changelog](https://canadaverse.org/beacon-dev/source.html) lists the complete review composition. @@ -105,7 +111,7 @@ Current sites: | [beacon-docs](https://github.com/MeshCore-Beacon/beacon-docs) | Shared contracts, operator guidance and this roadmap | `main` | | [beacon-mobile](https://github.com/MeshCore-Beacon/beacon-mobile) | Mobile client; coordinate API compatibility | `main` | -## Current observer-first release +## Observer-first release delivered for review The observer-first release is implemented in four focused review candidates: [server #169](https://github.com/MeshCore-Beacon/beacon-server/pull/169) (`91b21995`, closes #168), [web #79](https://github.com/MeshCore-Beacon/beacon-web/pull/79) (`3a0eb4c8`, closes #76), [web #80](https://github.com/MeshCore-Beacon/beacon-web/pull/80) (`b3a6b088`, closes #77) and [web #81](https://github.com/MeshCore-Beacon/beacon-web/pull/81) (`42ae09b7`, closes #78). All are out of draft. Server #169 follows #167; web order is #75 -> #79 -> #80 -> #81. Independent server #166 remains in the preview composition. Maintainers control acceptance and release; these issues remain open until their changes are accepted. @@ -119,7 +125,7 @@ See the [observer implementation and subsequent UX releases](app_documentation/o All ten original server/web contributions merged on September 24. The September 25 web batch is also accepted: #70 contains translations #63/#64/#65/#66/#69 plus Timestamp wording; #68 and #72 merged separately and closed #67/#71. The five earlier translation PRs were closed as included, with exact ancestry/tree equivalence verified. Web #73 and server #162/#163 then landed. That acceptance batch cleared the application queue. The new September 26 retention/endpoint PRs and docs #5 are now in review, as listed below. -Current accepted dev is server `dec643a2ade712cd60feb2bc761a5654c7c82714` / web `54b5093ac0302c7db9d51e1d7fe23570eae7cdb5`. Exact-head CI/image builds pass; server coverage/CodeQL pass and web CodeQL remains skipped. Stable releases are still server **v1.6.0** and web **v1.3.0**. The original September 24 freeze (`c02317a4` / `0f0a6ca5`) remains historical evidence, not proof that the newer server migrations are Pi-validated. +At the September 27 consolidation checkpoint, accepted dev was server `dec643a2ade712cd60feb2bc761a5654c7c82714` / web `54b5093ac0302c7db9d51e1d7fe23570eae7cdb5`. Exact-head CI/image builds pass; server coverage/CodeQL pass and web CodeQL remains skipped. Stable releases are still server **v1.6.0** and web **v1.3.0**. The original September 24 freeze (`c02317a4` / `0f0a6ca5`) remains historical evidence, not proof that the newer server migrations are Pi-validated. | Accepted PR | Scope | Merge commit | |---|---|---| @@ -154,7 +160,7 @@ Full Windows and native Pi Go/PostgreSQL checks pass, including rollback on arch Migration 039 preserved fingerprints of all 23 original application tables. The actual prior schema038 database, exact binary/configuration and private dump remain available for rollback; older pre-038 recovery is retained separately. Only the Beacon preview app restarted (about 33 seconds); 22 other containers were unchanged and both MQTT feeds reconnected. Public admin/backup and foreign detection remain disabled. [Current changelog and corresponding source](https://canadaverse.org/beacon-dev/source.html). -Current broader issues remain server #60 (admin), #72 (backup/import), #99 (packet summaries), #116 (MQTT investigation), and web #12 (remaining translations). Current priorities are the observer release and following investigation work described above. A measured month of accumulated history, production-scale capacity and physical Safari checks remain separate gates. Maintainers own stable releases and the owners handle production cutover. +Current broader issues remain server #60 (admin), #72 (backup/import), #99 (packet summaries), #116 (MQTT investigation), and web #12 (remaining translations). Current priorities are review feedback and server #181, followed by boundaries and the remaining investigation work. A measured month of accumulated history, production-scale capacity and physical Safari checks remain separate gates. Maintainers own stable releases and the owners handle production cutover. ## Delivered foundations @@ -184,16 +190,17 @@ The September 24 consolidation check built accepted server `c02317a4` and retain The September 20 #116 investigation has a new [current-build result](https://github.com/MeshCore-Beacon/beacon-server/issues/116#issuecomment-5753626821): a 600-second unmodified Pi capture kept both feeds connected and retained 2,169 new observations, with no ping timeout, disconnect, deadline, SQLSTATE error or HTTP 5xx response. App/PostgreSQL CPU averaged 2.14%/3.96% of one core. The preceding 3h39 log likewise has no MQTT loss or deadline error. Timestamp warnings were classified separately. This did not measure callback or pool-acquisition duration and does not establish the original cause or production capacity. No application, ordering, acknowledgement or service change was made; #116 remains open. Further capture should follow a recurrence or meaningful workload change, rather than repeatedly sampling the same healthy state. -1. **Review the current retention/endpoint and observer candidates.** Keep the ordered server #167 -> #169 -> #172 -> #174 and web #75 -> #79 -> #80 -> #81 -> #83 -> #85 -> #87 stacks; #166 is independent. Refresh with the existing workflow after acceptance. Maintainers choose the release breakpoint, versions, tags and main promotion. -2. **Connected investigation.** Connect packets, exact observed paths/routes, reporting observers and map actions with reliable Back navigation and visibly ambiguous identities. Continue focused issue #99/#12 work where it overlaps this accepted scope. -3. **Node/route/trace presentation, then distinct analytics questions and quality of life.** Follow the approved observer plan's subsequent releases; this phase does not claim full parity. -4. **Mesh Scopes interoperability.** Optional cached import is implemented in server #174; continue with consistent channel scope tags, using the [integration plan](app_documentation/mesh-scopes-plan.md). Retain manual names and separate observed/default/imported evidence. No API key is required; importer acceptance and release remain with maintainers. +1. **Reviews and listed issues first.** Fix [server #181](https://github.com/MeshCore-Beacon/beacon-server/issues/181), accepting valid numeric-offset packet timestamps with explicit regressions and unchanged time guards. Recheck feedback before starting; broad partial issues remain open. +2. **Accept the current queue in dependency order.** Server #167 -> #169 -> #172 -> #174 -> #176; independent #166. Web #75 -> #79 -> #80 -> #81 -> #83 -> #85 -> #87 -> #89 -> #92. Use the published-head table in the integration record. Maintainers choose merges, the release breakpoint, versions, tags and main promotion. +3. **Optional MeshMapper boundaries.** Scope import and channel tags are already implemented in #174/#176/#89. Next use the [boundary plan](app_documentation/meshmapper-boundaries-plan.md), preserving manual boundary priority, cached valid geometry and separate scope/forwarding evidence. Crossing analytics remain a later focused slice. +4. **Connected investigation and presentation.** Packet/route/observer links and return navigation are delivered for review. Continue node/trace presentation, distinct analytics questions and quality of life under the approved observer plan; address #99/#12 where the work overlaps. Full parity is not yet claimed. ## Listed work still open | Issue | Remaining scope | |---|---| -| [Server #116](https://github.com/MeshCore-Beacon/beacon-server/issues/116) | No recurrence in the September 20 retained log / ten-minute capture; still needs an attributable event with callback/pool timing | +| [Server #181](https://github.com/MeshCore-Beacon/beacon-server/issues/181) | Whole-second RFC3339 numeric-offset packet timestamps fall back to server time; add parser correction and regressions | +| [Server #116](https://github.com/MeshCore-Beacon/beacon-server/issues/116) | #179/#180 are integrated and pass bounded replay/route-lock checks; attributing the historical incident still requires matching evidence | | [Server #99](https://github.com/MeshCore-Beacon/beacon-server/issues/99) | Advert names and ACK/TRACE/PING references are accepted; define any remaining packet-type formats | | [Server #60](https://github.com/MeshCore-Beacon/beacon-server/issues/60) | Remaining administration/worker/persistence behavior; account records do not establish login sessions | | [Server #72](https://github.com/MeshCore-Beacon/beacon-server/issues/72) | Download #154 and archive validation #160 are accepted; import, browser access, deployment-file coverage and remote/scheduled backup remain | diff --git a/app_documentation/ingest-integration-20260928.md b/app_documentation/ingest-integration-20260928.md new file mode 100644 index 0000000..f33a408 --- /dev/null +++ b/app_documentation/ingest-integration-20260928.md @@ -0,0 +1,47 @@ +# Ingest integration validation — 28 September 2026 + +The Pi preview runs server `2ed2e03117f6c88795d446456e6d74c20c485d28` (tree `435400335edda9b9c0d1dc5210f5043403390c19`) and web `6b6884951a3dac01b592dfec83f0191879c5696c`. Accepted bases are server `db30c9b573357990c41166292e7cf42785c92cc4` and web `17f48fb932facbdcc45068afd2db6036ac2ec94d`. All fifteen application candidates remain included and out of draft. Published-head checks and both full-stack Check commands pass; web CodeQL remains skipped. Maintainers retain application merges, stable releases and production cutover. + +## Source integration + +Accepted server #177–#180 and web #91 are integrated through one coordinated helper refresh. The only source conflict was independent server #166: payload side effects must precede live endpoint resolution for inserted observations, while upstream's optional new-path repeat events must retain current identity without reapplying old adverts. Existing first/renamed-advert regressions failed on the upstream ordering and pass with the fix. A new repeat regression verifies fresh identity, no repeated side effects and no endpoint lookups for suppressed duplicates. + +| Repository / PR | Published head | Exact parent | +|---|---|---| +| [server #166](https://github.com/MeshCore-Beacon/beacon-server/pull/166) | `e1c4fa2f70ae6b5efc0054b91433460d07f06ce1` | `db30c9b573357990c41166292e7cf42785c92cc4` | +| [server #167](https://github.com/MeshCore-Beacon/beacon-server/pull/167) | `70c5c389f9886cb8d8c9d9d34462c805cad6fc6b` | `db30c9b573357990c41166292e7cf42785c92cc4` | +| [server #169](https://github.com/MeshCore-Beacon/beacon-server/pull/169) | `39c9d185087dcf6bc5aede8f8d1e86c61c1365c9` | `70c5c389f9886cb8d8c9d9d34462c805cad6fc6b` | +| [server #172](https://github.com/MeshCore-Beacon/beacon-server/pull/172) | `0fc1398eecaa24f1dbdd0fae28bc259f15a439f2` | `39c9d185087dcf6bc5aede8f8d1e86c61c1365c9` | +| [server #174](https://github.com/MeshCore-Beacon/beacon-server/pull/174) | `235e2e0fd672febf8e236d5b5859ef3d660028df` | `0fc1398eecaa24f1dbdd0fae28bc259f15a439f2` | +| [server #176](https://github.com/MeshCore-Beacon/beacon-server/pull/176) | `cf8cee332a6494a2ab603eab306883fc6ba17f8d` | `235e2e0fd672febf8e236d5b5859ef3d660028df` | +| [web #75](https://github.com/MeshCore-Beacon/beacon-web/pull/75) | `e65e8e42ec7bcd75029ebc636645eedc2b889cbf` | `17f48fb932facbdcc45068afd2db6036ac2ec94d` | +| [web #79](https://github.com/MeshCore-Beacon/beacon-web/pull/79) | `4f3819b805ff8293939d1e6fb3918f05cafaadd5` | `e65e8e42ec7bcd75029ebc636645eedc2b889cbf` | +| [web #80](https://github.com/MeshCore-Beacon/beacon-web/pull/80) | `04ca3da919d9b0430f364a1a93bf36cf56e9770a` | `4f3819b805ff8293939d1e6fb3918f05cafaadd5` | +| [web #81](https://github.com/MeshCore-Beacon/beacon-web/pull/81) | `2f76286ec89c81db0345937b8c1712f363bd2e43` | `04ca3da919d9b0430f364a1a93bf36cf56e9770a` | +| [web #83](https://github.com/MeshCore-Beacon/beacon-web/pull/83) | `0787e5b5fdb1d8d47aefcbb20e816ba280a2ebe2` | `2f76286ec89c81db0345937b8c1712f363bd2e43` | +| [web #85](https://github.com/MeshCore-Beacon/beacon-web/pull/85) | `b21b8704c7403d7f963dc445de7137b9857e194e` | `0787e5b5fdb1d8d47aefcbb20e816ba280a2ebe2` | +| [web #87](https://github.com/MeshCore-Beacon/beacon-web/pull/87) | `be5434a3d4784e26434b5395693896dc7139fd4d` | `b21b8704c7403d7f963dc445de7137b9857e194e` | +| [web #89](https://github.com/MeshCore-Beacon/beacon-web/pull/89) | `037858b4646349c4599d9067fb8f42ad9d1a34a3` | `be5434a3d4784e26434b5395693896dc7139fd4d` | +| [web #92](https://github.com/MeshCore-Beacon/beacon-web/pull/92) | `6b6884951a3dac01b592dfec83f0191879c5696c` | `037858b4646349c4599d9067fb8f42ad9d1a34a3` | + +## Validation and limits + +- Local formatting/build/vet/tests and exact-head GitHub CI pass for all server candidates; local web build/lint/tests and exact-head build checks pass for all nine web candidates. +- The composed native Pi server build, full PostgreSQL suite and new queue/order/memory/drain/cancellation/clock-skew/route-lock tests pass. Linux CI supplies race checks; Pi ThreadSanitizer cannot run with this host's VMA layout. +- A native fixture used real PostgreSQL storage, the actual callback/ingest queue, 504 scope candidates, encryption/decryption, and normal plus opt-in live consumers. A later route-maintenance batch was deliberately blocked while an earlier completed batch continued ingesting. Its 3,200 inputs produced exactly 100 packets, 800 stored observations, 100 decrypted messages, 800 normal events, 2,400 opt-in events and 3,200 acknowledgements, with zero drops/invalid inputs in 12.217 seconds. The fixture used an isolated test schema and published no RF or broker traffic. It does not prove universal production losslessness or explain the historical 1,810-counter discrepancy. +- Native web build/lint and all 935 tests pass. All 18 rebuilt asset files are byte-identical to the previously geometry-tested build; exact corresponding source is newly offered. Public scoped packets, decoded Public history and message-to-packet inspection pass. English desktop and French 390px phone smoke checks pass without horizontal overflow or browser console errors. Existing untranslated controls remain under web #12. +- Both real MQTT feeds advanced across a 62.76-second observation window, with zero restarts and no queue-overflow/database errors or disconnects. Input warnings remain: 24 malformed-region messages, 51 timestamp clamps and 38 timestamp parsing fallbacks in that window. The numeric-offset timestamp parser gap is tracked as [server #181](https://github.com/MeshCore-Beacon/beacon-server/issues/181); do not call these logs clean. + +## Deployment and recovery + +Only the Beacon app restarted; the other 22 containers were unchanged. Frontend publication changed no container identity. Configuration digest remains `c09c3aea14eef795baf84d0462dd53c59de5f6e432a6023f93592f5adf6cc277`, runtime mode 0644, schema042, 72h packets / 30d summaries / 720h telemetry, standard Public key, 504 candidates and the YOW importer. Public admin/backup/foreign detection remain disabled. The country-wide candidate list is still a snapshot, not an automatic discovery service. + +A fresh 58,002,666-byte database dump restored successfully and has a private checksum-verified off-Pi copy. SHA-256: `b6149d551db14a7bc1e847fd9c1bf19aab77e065e6d25680703082155f89612b`. Its disposable verification database was removed after the successful restore; the dump and original databases remain. Immediate same-schema rollback preserves new rows and restores server `7c9599b1` / web `dfeb2777`, using backup `ingest-cutover-20260928T222830Z` and Pi `evidence/ingest-integration-20260928/deploy-server.py rollback`. Frontend-only recovery uses backup `web-20260928T224458Z` and the deployment helper in that evidence directory, paired with server `2ed2e031`. Older schema041 recovery remains separately paired; do not apply older rollback commands directly to this build. + +The [public source/changelog](https://canadaverse.org/beacon-dev/source.html) identifies the exact served revisions. Server binary SHA-256: `c3150945e53b49d7549a1936720f971e4c8c4ee9c2e26739184fd37f5748ee6e`; server source: `efe28b8cab9d87be7e2bd13e697bab060325e2840d75f22f60199bb61fa94824`; web corresponding source: `ccfa5750b606ca6c9506af1eb8fc838656efebd8f9e672333ec29e33a8f64f8f`. + +## Next work + +Review feedback and [server #181](https://github.com/MeshCore-Beacon/beacon-server/issues/181) take priority. Add parser regression coverage and accept supported RFC3339 numeric-offset timestamps without weakening invalid/future/stale guards. Broad server #60/#72/#99/#116 and web #12 remain open. Optional MeshMapper boundaries follow issue work. + +Both application repositories still have merge commits disabled and the contributor has READ access. A maintainer must enable that option; no repository setting was changed. Merge commits preserve parent ancestry, but source conflicts and validation remain. Keep this queue in dependency order, use the existing helper for current composition/history repair, and start independent work from fresh dev after the queue is accepted. From f556f0aea6ad382d95a8ee13feed35a9da55ac3b Mon Sep 17 00:00:00 2001 From: n30nex Date: Tue, 29 Sep 2026 19:44:43 -0400 Subject: [PATCH 35/69] docs: record review corrections and retained-history release candidate --- CONTRIBUTOR_WORKFLOW.md | 6 +- RELEASE-CHECKLIST.md | 14 +++-- ROADMAP.md | 28 ++++++--- app_documentation/channel-scope-metadata.md | 16 +++++ app_documentation/meshmapper-scope-import.md | 26 ++++++++ app_documentation/review-release-20260929.md | 65 ++++++++++++++++++++ 6 files changed, 140 insertions(+), 15 deletions(-) create mode 100644 app_documentation/channel-scope-metadata.md create mode 100644 app_documentation/meshmapper-scope-import.md create mode 100644 app_documentation/review-release-20260929.md diff --git a/CONTRIBUTOR_WORKFLOW.md b/CONTRIBUTOR_WORKFLOW.md index 1c4e469..3b2ce0a 100644 --- a/CONTRIBUTOR_WORKFLOW.md +++ b/CONTRIBUTOR_WORKFLOW.md @@ -30,11 +30,11 @@ A source conflict still needs review. The helper automates routine history movem ## Current integration example -The 28 September coordinated refresh integrates server dev `db30c9b5` (#177–#180) and web dev `17f48fb9` (#91). Server #166 stays independent; #167 -> #169 -> #172 -> #174 -> #176 is the ordered server chain. Web #75 -> #79 -> #80 -> #81 -> #83 -> #85 -> #87 -> #89 -> #92 remains one declared chain. Only #166 had a source conflict; it was resolved with advert/repeat regressions before publication. Both full-stack Check commands pass for all fifteen exact published heads. +The September 29 review pass preserves the accepted dev bases and updates each affected server PR through the existing helper. Two authored merges kept both the new observer documentation and route section, and both query groups in queries.sql. SQLc/Swagger outputs were regenerated. Server #166/#184 remain independent; the ordered chain is #167 -> #169 -> #172 -> #174 -> #176. Web #95 is a new child of #92; none of the nine existing web parents needed rebasing. -The Pi runs composed server `2ed2e031` / web `6b688495`, retaining every current feature. Native database/queue/route-lock checks and all 935 web tests pass. All 18 rebuilt web assets are byte-identical to the prior geometry-tested build; the corresponding source offer identifies the actual refreshed source. Same-schema rollback preserves new rows. See [validation and exact PR heads](app_documentation/ingest-integration-20260928.md). +Both full-stack Check commands pass for all seventeen actual application heads. The Pi runs server a35cba1d / web e1133ab5 with validated database repair/rollback and 940 web tests. See [current heads and evidence](app_documentation/review-release-20260929.md). All candidates are out of draft; requested-change reviews still need maintainer re-review. -Merge commits remain disabled in both application repos, and the contributor has READ access. A maintainer must enable that option. This integration changes no merge policy or helper algorithm; it uses the existing isolated conflict stop, cached validation, leased publication and exact-head checks. Native tests and public/runtime checks supplement CI when combined behavior changes. +Merge commits remain disabled and contributor permission remains READ. A maintainer must enable that repository option. No helper algorithm or repository policy changed in this phase. Continue from fresh dev after acceptance and stack only real dependencies. ## Setup diff --git a/RELEASE-CHECKLIST.md b/RELEASE-CHECKLIST.md index 1164afd..f33858b 100644 --- a/RELEASE-CHECKLIST.md +++ b/RELEASE-CHECKLIST.md @@ -1,14 +1,18 @@ # Server/web consolidation release -Status: development preview and review handoff, 28 September 2026. Stable tags and production cutover remain owner-managed. This is not a complete CoreScope parity claim. +Status: development preview and review handoff, 29 September 2026 (Toronto). Stable tags and production cutover remain owner-managed. This is not a complete CoreScope parity claim. -## Current integration gate — 28 September +## Current integration gate — 29 September -The Pi preview now runs server `2ed2e031` / web `6b688495`, integrating upstream server #177–#180 and web #91 with all fifteen pending candidates. Both queues pass current-base Check and actual published-head CI (web CodeQL skipped). Full native server/PostgreSQL and 935 web tests pass. The isolated 3,200-input/504-scope replay preserves expected rows and events under route-lock contention with zero drops; both actual inputs advanced during a 63-second observation. This does not establish universal lossless production ingestion or explain the historical counter difference. +All six server reviews are addressed in their existing PR sequence. The fixes restore analytics indexes, consolidate unmerged migrations, preserve current partial activity buckets, align cache windows, narrow the route index, keep manual scope priority and simplify channel insertion metadata. Independent server #184 fixes RFC3339 offsets; new web #95 follows #92 and matches time choices to retained data. Existing candidates remain included. -Fresh source/18-asset verification and desktop/French-phone smoke checks pass. Only the Beacon app restarted; other services and schema042/configuration stayed intact. A fresh dump restored and was verified off-Pi; current and rollback binary/frontend/source remain. See [exact heads, evidence and recovery](app_documentation/ingest-integration-20260928.md). +The Pi runs server `a35cba1d` / web `e1133ab5`. All seventeen published application heads pass Check/CI (web CodeQL skipped); native Go/PostgreSQL and Windows/Pi web build/lint/all 940 tests pass. The restored-copy repair preserved raw rows and archive fingerprints, and rollback index restoration passed. A 3,200-input/504-scope replay had all expected rows/events and zero fixture drops. The one-minute live sample had no parser fallbacks, queue overflows, SQL errors, restarts or reconnects; malformed-IATA and clock-skew warnings remain. -Remaining before further feature work: review feedback, then [server #181](https://github.com/MeshCore-Beacon/beacon-server/issues/181) timestamp parsing. Input timestamp/region warnings remain. Physical iPhone Safari, production-volume capacity, broad partial issues and maintainer release acceptance remain distinct gates. Enabling merge commits requires a maintainer; the contributor's account is READ. +Visible periods are **24h / 3d** for observer monitoring and route evidence, and **24h / 3d / 30d** for summary-backed Analytics. Seven-day buttons are removed. Raw comparison spans are capped at three days. Durable hourly aggregates already preserve expired packet counts; materialized views combine them with live rows. Older summaries accumulate after archiving starts, and packet detail remains unavailable after expiry. Revised labels and notes are English/French. + +Current acceptance still requires maintainer re-review, especially #167/#169/#174. No upstream merge, stable release or production cutover was performed. The next focused issue is server #183 (saved-route prefix-width changes); broad partial issues remain open. External server #182 and web #93 are unmerged and not in this tested composition. Owners decide the release breakpoint and production switch. + +See [review corrections, source heads and recovery](app_documentation/review-release-20260929.md). No stable release is claimed until maintainers accept the reviewed application heads and choose the release artifacts. ## Packet reception investigation — 27 September diff --git a/ROADMAP.md b/ROADMAP.md index 41de3cf..ac9da39 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,12 +1,24 @@ # Beacon parity and analytics roadmap -Updated 28 September 2026 UTC. This is the working roadmap for n30nex's ongoing contributions toward CoreScope feature parity. Maintainers decide acceptance and merge order; deployment owners handle the production switch. +Updated 29 September 2026 (Toronto). This is the working roadmap for n30nex's ongoing contributions toward CoreScope feature parity. Maintainers decide acceptance and merge order; deployment owners handle the production switch. Refresh GitHub issues, PR feedback and branch state before starting a phase. This document is a snapshot, and linked issues/PRs are the current source of truth. Release-check correction, 21 September UTC: the workflow now includes independent preview PRs in Status and Check, applies the same CI/head/fork/target requirements to them, and rechecks the prepared independent inputs before publication. This covers packet summaries #161 and map correction #61 without adding them to the ordered stacks. The backup CLI #160 retains its separate check. Twenty-three offline regressions cover these gates and the existing no-rebase/cache behavior. See [the contributor workflow](CONTRIBUTOR_WORKFLOW.md). -Latest integrated upstream is server `db30c9b5` (#177–#180) and web `17f48fb9` (#91). One coordinated Refresh/Publish/Check pass is complete. The Pi now runs composed server `2ed2e031` / web `6b688495`, including all fifteen review candidates; use the current heads below rather than historical checkpoint SHAs. Recheck upstream before new work. +Accepted and integrated bases remain server `db30c9b5` / web `17f48fb9`. The current tested preview is server `a35cba1d` / web `e1133ab5`, with all seventeen application review candidates. The sections below retain dated historical checkpoints; use the September 29 record for current heads and rollback. + +## Review corrections and history windows — 29 September + +All six server reviews are addressed in their existing PR sequence. The fixes restore analytics indexes, consolidate unmerged migrations, preserve current partial activity buckets, align cache windows, narrow the route index, keep manual scope priority and simplify channel insertion metadata. Independent server #184 fixes RFC3339 offsets; new web #95 follows #92 and matches time choices to retained data. Existing candidates remain included. + +The Pi runs server `a35cba1d` / web `e1133ab5`. All seventeen published application heads pass Check/CI (web CodeQL skipped); native Go/PostgreSQL and Windows/Pi web build/lint/all 940 tests pass. The restored-copy repair preserved raw rows and archive fingerprints, and rollback index restoration passed. A 3,200-input/504-scope replay had all expected rows/events and zero fixture drops. The one-minute live sample had no parser fallbacks, queue overflows, SQL errors, restarts or reconnects; malformed-IATA and clock-skew warnings remain. + +Visible periods are **24h / 3d** for observer monitoring and route evidence, and **24h / 3d / 30d** for summary-backed Analytics. Seven-day buttons are removed. Raw comparison spans are capped at three days. Durable hourly aggregates already preserve expired packet counts; materialized views combine them with live rows. Older summaries accumulate after archiving starts, and packet detail remains unavailable after expiry. Revised labels and notes are English/French. + +Current acceptance still requires maintainer re-review, especially #167/#169/#174. No upstream merge, stable release or production cutover was performed. The next focused issue is server #183 (saved-route prefix-width changes); broad partial issues remain open. External server #182 and web #93 are unmerged and not in this tested composition. Owners decide the release breakpoint and production switch. + +[Exact current heads, validation and recovery](app_documentation/review-release-20260929.md). ## Ingest integration delivered — 28 September @@ -14,9 +26,9 @@ The upstream ingest queue and bounded route-reconfirmation changes are integrate All fifteen published application heads pass checks (web CodeQL remains skipped). The composed native Pi server/PostgreSQL suite passes. With 504 scope candidates and blocked route maintenance, an isolated 3,200-input replay preserved the expected 100 packets, 800 observations, 100 decrypted messages and 2,400 opt-in live events with zero queue drops. Native web build/lint and all 935 tests pass. All 18 public assets and both source archives match; English desktop, French phone, scoped packet rows, Public history and packet inspection pass. This is bounded fixture evidence, not proof of universal production losslessness. -Both real MQTT feeds advanced during the 63-second runtime observation with no restarts or queue-overflow errors. Malformed region and timestamp warnings remain. Valid whole-second timestamps with numeric UTC offsets expose a pre-existing parser gap, now [server #181](https://github.com/MeshCore-Beacon/beacon-server/issues/181). Fix that next, after checking new reviews/issues; then resume optional MeshMapper boundaries. The older CoreScope legacy-counter gap remains unexplained. +Both real MQTT feeds advanced during the 63-second runtime observation with no restarts or queue-overflow errors. Malformed region and timestamp warnings remain. Valid whole-second timestamps with numeric UTC offsets expose a pre-existing parser gap, now [server #181](https://github.com/MeshCore-Beacon/beacon-server/issues/181). That parser correction is now in #184; see the September 29 record for current priority. The older CoreScope legacy-counter gap remains unexplained. -Current preview: server `2ed2e03117f6c88795d446456e6d74c20c485d28` / web `6b6884951a3dac01b592dfec83f0191879c5696c`. Configuration, schema042, Public key, 504 exact-case candidates, YOW importer and 72h/30d/720h retention are unchanged. A fresh dump restored successfully and was checksum-verified off the Pi. Only Beacon restarted; the other 22 containers were unchanged. Same-schema rollback retains new data and restores server `7c9599b1` / web `dfeb2777` from `ingest-cutover-20260928T222830Z`; frontend-only recovery is `web-20260928T224458Z` paired with the new backend. Exact procedures and current PR heads are in the integration record below. +September 28 checkpoint: server `2ed2e03117f6c88795d446456e6d74c20c485d28` / web `6b6884951a3dac01b592dfec83f0191879c5696c`. Configuration, schema042, Public key, 504 exact-case candidates, YOW importer and 72h/30d/720h retention are unchanged. A fresh dump restored successfully and was checksum-verified off the Pi. Only Beacon restarted; the other 22 containers were unchanged. Same-schema rollback retains new data and restores server `7c9599b1` / web `dfeb2777` from `ingest-cutover-20260928T222830Z`; frontend-only recovery is `web-20260928T224458Z` paired with the new backend. Exact procedures and current PR heads are in the integration record below. The shared UTF-8 corrections and merge guidance remain in place. Both application repos still disable merge commits; the contributor has READ access, so a maintainer must enable that setting. The helper remains optional for maintainers and unrelated to ingestion. Owners retain upstream merges and production release. @@ -190,8 +202,8 @@ The September 24 consolidation check built accepted server `c02317a4` and retain The September 20 #116 investigation has a new [current-build result](https://github.com/MeshCore-Beacon/beacon-server/issues/116#issuecomment-5753626821): a 600-second unmodified Pi capture kept both feeds connected and retained 2,169 new observations, with no ping timeout, disconnect, deadline, SQLSTATE error or HTTP 5xx response. App/PostgreSQL CPU averaged 2.14%/3.96% of one core. The preceding 3h39 log likewise has no MQTT loss or deadline error. Timestamp warnings were classified separately. This did not measure callback or pool-acquisition duration and does not establish the original cause or production capacity. No application, ordering, acknowledgement or service change was made; #116 remains open. Further capture should follow a recurrence or meaningful workload change, rather than repeatedly sampling the same healthy state. -1. **Reviews and listed issues first.** Fix [server #181](https://github.com/MeshCore-Beacon/beacon-server/issues/181), accepting valid numeric-offset packet timestamps with explicit regressions and unchanged time guards. Recheck feedback before starting; broad partial issues remain open. -2. **Accept the current queue in dependency order.** Server #167 -> #169 -> #172 -> #174 -> #176; independent #166. Web #75 -> #79 -> #80 -> #81 -> #83 -> #85 -> #87 -> #89 -> #92. Use the published-head table in the integration record. Maintainers choose merges, the release breakpoint, versions, tags and main promotion. +1. **Reviews and listed issues first.** The #181 correction is submitted as #184. Next address [server #183](https://github.com/MeshCore-Beacon/beacon-server/issues/183), preserving honest saved-route evidence when prefix widths change. Recheck maintainer feedback first; broad partial issues remain open. +2. **Accept the current queue in dependency order.** Server #167 -> #169 -> #172 -> #174 -> #176; independent #166. Web #75 -> #79 -> #80 -> #81 -> #83 -> #85 -> #87 -> #89 -> #92 -> #95. Independent server #184 follows dev. Use the published-head table in the integration record. Maintainers choose merges, the release breakpoint, versions, tags and main promotion. 3. **Optional MeshMapper boundaries.** Scope import and channel tags are already implemented in #174/#176/#89. Next use the [boundary plan](app_documentation/meshmapper-boundaries-plan.md), preserving manual boundary priority, cached valid geometry and separate scope/forwarding evidence. Crossing analytics remain a later focused slice. 4. **Connected investigation and presentation.** Packet/route/observer links and return navigation are delivered for review. Continue node/trace presentation, distinct analytics questions and quality of life under the approved observer plan; address #99/#12 where the work overlaps. Full parity is not yet claimed. @@ -199,7 +211,9 @@ The September 20 #116 investigation has a new [current-build result](https://git | Issue | Remaining scope | |---|---| -| [Server #181](https://github.com/MeshCore-Beacon/beacon-server/issues/181) | Whole-second RFC3339 numeric-offset packet timestamps fall back to server time; add parser correction and regressions | +| [Server #181](https://github.com/MeshCore-Beacon/beacon-server/issues/181) | Fixed by #184; awaits maintainer acceptance | +| [Server #183](https://github.com/MeshCore-Beacon/beacon-server/issues/183) | Saved-route hash-prefix metadata can lag a change of width; next focused issue | +| [Web #94](https://github.com/MeshCore-Beacon/beacon-web/issues/94) | Corrected periods/labels in #95; awaits acceptance | | [Server #116](https://github.com/MeshCore-Beacon/beacon-server/issues/116) | #179/#180 are integrated and pass bounded replay/route-lock checks; attributing the historical incident still requires matching evidence | | [Server #99](https://github.com/MeshCore-Beacon/beacon-server/issues/99) | Advert names and ACK/TRACE/PING references are accepted; define any remaining packet-type formats | | [Server #60](https://github.com/MeshCore-Beacon/beacon-server/issues/60) | Remaining administration/worker/persistence behavior; account records do not establish login sessions | diff --git a/app_documentation/channel-scope-metadata.md b/app_documentation/channel-scope-metadata.md new file mode 100644 index 0000000..c88aaae --- /dev/null +++ b/app_documentation/channel-scope-metadata.md @@ -0,0 +1,16 @@ +# Channel-message transport scope + +Channel history, the global/hash message APIs, reconnect backfill and `channelMessage` events now include `scope` and `scopeStatus`. Live events also include their stored message `id`; existing fields, endpoints and cursors are unchanged. + +| Status | Meaning | +|---|---| +| `matched` | `scope` names the scope recorded on the first stored packet. | +| `unscoped` | The first stored packet has no transport codes; `scope` is null. | +| `unknown` | Transport codes were recorded but no scope name was resolved, including ambiguous matches. | +| `unavailable` | Scope/capture metadata is missing. | + +These labels belong to the first stored packet, not every later reception of the same payload. Later paths may differ. Live delivery reads that same evidence while inserting the channel message, so a later reception cannot relabel it differently from history. Existing scope query filters retain their meaning. Imported catalogue membership and channel names are never used as message evidence. + +The insertion query returns metadata only for a new message. Duplicate deliveries still produce no extra live message. Adding a decryption key later uses the existing backfill path, retains the stored packet evidence and does not broadcast old messages as new traffic. This change adds no keys, migration or historical scan. Expired packet/message detail remains unavailable. + +Deploy this API before its web consumer. Older clients may ignore the added fields; clients reading older servers must treat absent metadata as unavailable, not assume a message is unscoped. Scope filtering of live events requires the new fields. The channel encryption key and the packet transport scope are separate concepts. diff --git a/app_documentation/meshmapper-scope-import.md b/app_documentation/meshmapper-scope-import.md new file mode 100644 index 0000000..3d16ef8 --- /dev/null +++ b/app_documentation/meshmapper-scope-import.md @@ -0,0 +1,26 @@ +# Optional MeshMapper scope discovery + +Beacon can periodically read the [public Scopes API](https://wiki.meshmapper.net/scopes-api/) to discover additional transport scope names for configured regions. Manual `scopes` remain authoritative. This does not supply channel decryption keys or per-repeater records. + +```yaml +meshmapper: + scopes: + enabled: true + refresh_interval: 1h + sources: + YOW: https://yow.meshmapper.net/get_scopes.php +``` + +The IATA must already occur in `regions[].iatas`. Set the exact published regional HTTPS URL; credentials, query strings, other hosts and redirects are rejected. The response must identify the same region and exactly one zone. Group catalogues are rejected because they cannot assign each scope/count to individual member regions. No API key is required. + +The default is disabled. Refresh defaults to one hour and accepts 5 minutes through 24 hours. One source is considered every 15 seconds, keeping requests below four per minute per process. Multiple deployments share MeshMapper's per-IP allowance. ETags reduce transfer, HTTP 304 advances the last successful check, and 429 pauses all sources until the later of the normal retry time and `Retry-After`. The pause survives restarts. HTTP 503 also honors that source's `Retry-After`. + +Responses are limited to 64 KiB and 64 names per source, with at most 16 sources. Names are case-sensitive and at most 128 bytes before Beacon's usual `#` normalization; existing `#`/`$` prefixes are retained. Zero-count monitored names are kept. Imported matching candidates are limited to their source IATA; manual keys keep their existing global applicability. Manual keys keep their existing first-match priority, including when an imported name has the same short code. Multiple matching imported names remain unresolved when no manual key matches. + +Migration 042 stores the last validated response, validator and check/retry timestamps in PostgreSQL. Saving a response and its scope identities is atomic. Timeouts, invalid responses and HTTP errors retain the prior catalogue; startup restores it without HTTP. A successful empty response removes that source's active imported membership. Removing or disabling a source takes effect on restart. Scope identities referenced by historical records are retained. Adding an imported name to the manual list promotes it to manual ownership. + +Operator logs under `component=meshmapper.scopes` show the IATA, source, active name count, source generation time, last successful check, next attempt and last error. An empty last error means the latest request succeeded; a zero checked time means no successful response has been stored. A 304 retains the original source generation time because MeshMapper excludes that timestamp from its ETag. Cached names may be old during an outage; use the check/error fields to assess freshness. + +Catalogue counts remain source metadata. They never create Beacon packet counts, observer associations or node defaults. Actual packet-code matching supplies those associations through the existing ingest path. New names affect subsequent packets; no historical scan is started. Channel message tags and import-status UI are separate follow-ups. An invalid saved catalogue is logged and excluded until a valid refresh; other sources and manual keys still load. Database access failures remain startup errors. Disabling the importer restores manual-only matching. Previously imported identities remain visible in `/scopes` and historical records; visibility does not imply an active matcher candidate. + +Validate and back up PostgreSQL before applying the migration. Rollback to an older server must restore its matching pre-import database and configuration as well as its binary; an older matcher does not understand imported-only ownership or regional candidate limits. diff --git a/app_documentation/review-release-20260929.md b/app_documentation/review-release-20260929.md new file mode 100644 index 0000000..9ac1cf6 --- /dev/null +++ b/app_documentation/review-release-20260929.md @@ -0,0 +1,65 @@ +# Review corrections and retained-history periods — 29 September 2026 + +The Pi preview runs server `a35cba1d2c5cd1dccfa4a99b41717747ebd2f2fc` and web `e1133ab5c3364349af900fd4f2093384fbf34f56`, with all prior candidates and the new server #184 / web #95 included. Accepted bases remain server `db30c9b5` and web `17f48fb9`. Both stack Check commands pass for all seventeen exact published application heads; web CodeQL remains skipped. No upstream merge, stable tag or production cutover was performed. + +## Maintainer review disposition + +| PR | Implemented response | +|---|---| +| Server #166 | Keeps upstream repeat delivery, resolves current advert identity after side effects, removes the redundant observation stub override. Existing inserted/renamed/repeat regressions pass. The optional additional UpsertNode-ID optimization is deferred; repeat events still need current stored identity. | +| Server #167 | Restores both hourly Signal/Paths indexes; folds unknown observer activity into migration 039 and removes the need for a second observer-view rebuild. Archive counts have NOT NULL/default 0, cleanup runners serialize before shared upserts, and hourly IATA summaries explicitly cover 30 UTC days. | +| Server #169 | Removes unmerged 040, preserves the current partial bucket on live requests, aligns fixed-end cache keys and moves the summary query into queries.sql. Freshness fields are explicitly measured at generatedAt even with until; recordedPackets follows the activity window. Doc comments are corrected. | +| Server #172 | Narrows the concurrent evidence index and query with literal hop_count >= 2, repairs unrelated encoding, and tests index contents and custom/generic plans. Prefix-width drift is tracked separately in #183. | +| Server #174 | Manual matches retain first-match priority; ambiguity applies among imported names. Snapshot reads allocate no catalogue copy, invalid saved catalogues do not stop manual startup, and parent cancellation is normal shutdown. Queries/import grouping are tidied. Documentation explains that historical imported identities remain visible after disabling matching. | +| Server #176 | Operator documentation moves out of generated Swagger output, insertion comments are corrected, Message replaces the redundant NewMessage flag, and WebSocket scopeStatus has the enum tag. Contracts are regenerated on the combined stack. | + +The coordinated refresh needed two reviewed merges: preserving the new observer explanation beside route documentation, and keeping both observer-summary and catalogue queries in queries.sql. Generated SQLc/Swagger output was regenerated. Existing feature boundaries and all web parents were preserved; web #95 is a new child of #92. + +## User-approved history choices + +- Observer monitoring and route evidence offer 24h/3d. Explicit raw route/comparison periods cannot exceed 72 hours. Old rolling observer/route 7d/30d links normalize to 3d; copied observer links use that effective range. Explicit invalid timestamps are not silently changed. +- Summary-backed Analytics offers 24h/3d/30d, without seven-day buttons. Traffic, payload, top-observer/advertiser/talker, Signal and Paths summaries survive raw expiry via the durable hourly archive in server #167, combined with live rows by materialized views. A materialized view alone would lose that data when refreshed after deletion. +- Scopes/population/clock/graph views keep their existing non-windowed semantics. Packet detail and exact overlap still depend on retained raw records. Older summaries accumulate after archiving starts; already-purged data cannot be recovered. +- The latest activity bucket may be partial. Revised range/help text ships in English and French. The underlying policy remains 72h raw packets / 30d summaries / 720h telemetry. + +## Validation + +The complete native Go/PostgreSQL suite passes, including archive failure/retry/concurrency, unknown payloads, index contents and recovery, cache windows, immutable scope snapshots, catalogue restart/cancellation, channel duplicate behavior and timestamp format/skew cases. Fresh migrations include the corrected 039 and omit 040; the existing preview preserves its earlier 040 journal entry and receives a separate operator repair. + +The restored preview repair retained 245,934 packets, 919,654 observations, 4,263 messages, 1,956 nodes and 19,861 routes, with identical archive fingerprints. All three affected indexes and 16 count constraints validated. Restoring the old broader route index also passed on that copy. The live repair removed no data; ingestion advanced during its 10.95-second execution. Only the Beacon app restarted for deployment; 23 other containers were unchanged. Frontend publication changed no containers. + +An isolated real-PostgreSQL replay with 504 scope candidates and a blocked later route-maintenance batch preserved 100 packets, 800 observations, 100 decrypted messages, 800 normal events, 2,400 opt-in events and 3,200 acknowledgements from 3,200 inputs, with zero fixture drops/invalid inputs in 11.481 seconds. This does not prove universal production losslessness or explain the old 1,810-counter gap. + +During a 60.61-second live sample both feeds advanced with zero restarts, parser fallbacks, queue overflows, SQL errors or reconnects. There were 59 clock-skew clamps and 14 malformed-IATA warnings. These remaining input problems are not described as clean logs or proven packet loss. + +Windows and native Pi web build/lint and all 940 tests pass. Desktop and French 390px phone checks load real Pi data, show the intended period controls and fit without horizontal overflow. Background-browser clipboard reads were unavailable; canonical effective-range generation is covered by regressions. Physical iPhone Safari remains a separate owner check. All 18 public assets and both source archives match the [changelog/source offer](https://canadaverse.org/beacon-dev/source.html). + +## Review and release gates + +All seventeen application candidates are out of draft. Server #167/#169/#174 had requested changes and require renewed maintainer review; passing checks do not dismiss those reviews. Server #184 closes #181 and web #95 closes #94 on acceptance. Existing focused closure references remain. Broad server #60/#72/#99/#116 and web #12 stay open. Route prefix-width drift #183 is the next focused issue; optional boundaries and wider parity follow issue work. + +External server #182 (private CPU profiling) and web #93 (node-to-map navigation) remain separate unmerged work and are not included in this tested composition. Maintainers decide their integration and the release breakpoint. Both application repos still disable merge commits and the contributor has READ access; enabling that option requires a maintainer. Production-volume evidence, a month of accumulated summary history, physical Safari and owner acceptance remain separate gates; this is a review candidate, not a full parity or production release claim. + +| Repository / PR | Published head | Parent | +|---|---|---| +| [server #166](https://github.com/MeshCore-Beacon/beacon-server/pull/166) | `319df7da5246a71f9693f923d2124bf831eabb82` | `db30c9b573357990c41166292e7cf42785c92cc4` | +| [server #167](https://github.com/MeshCore-Beacon/beacon-server/pull/167) | `2ab36356fa023abb247db45c8a33de4410507f64` | `db30c9b573357990c41166292e7cf42785c92cc4` | +| [server #169](https://github.com/MeshCore-Beacon/beacon-server/pull/169) | `8df398ab3da6d65ccb8db31efcbdc599976b582a` | `2ab36356fa023abb247db45c8a33de4410507f64` | +| [server #172](https://github.com/MeshCore-Beacon/beacon-server/pull/172) | `21ea989162659e6efc6dd7cd3a904a2eef655401` | `8df398ab3da6d65ccb8db31efcbdc599976b582a` | +| [server #174](https://github.com/MeshCore-Beacon/beacon-server/pull/174) | `e27fcb3783a2ae2c7abccb22edaff0195caea348` | `21ea989162659e6efc6dd7cd3a904a2eef655401` | +| [server #176](https://github.com/MeshCore-Beacon/beacon-server/pull/176) | `42d50bc692c53a593a1cd65945154e842a72c369` | `e27fcb3783a2ae2c7abccb22edaff0195caea348` | +| [server #184](https://github.com/MeshCore-Beacon/beacon-server/pull/184) | `c39df3043931dbe37d51a73d15f050a1c61d1a3b` | `db30c9b573357990c41166292e7cf42785c92cc4` | +| [web #75](https://github.com/MeshCore-Beacon/beacon-web/pull/75) | `e65e8e42ec7bcd75029ebc636645eedc2b889cbf` | `17f48fb932facbdcc45068afd2db6036ac2ec94d` | +| [web #79](https://github.com/MeshCore-Beacon/beacon-web/pull/79) | `4f3819b805ff8293939d1e6fb3918f05cafaadd5` | `e65e8e42ec7bcd75029ebc636645eedc2b889cbf` | +| [web #80](https://github.com/MeshCore-Beacon/beacon-web/pull/80) | `04ca3da919d9b0430f364a1a93bf36cf56e9770a` | `4f3819b805ff8293939d1e6fb3918f05cafaadd5` | +| [web #81](https://github.com/MeshCore-Beacon/beacon-web/pull/81) | `2f76286ec89c81db0345937b8c1712f363bd2e43` | `04ca3da919d9b0430f364a1a93bf36cf56e9770a` | +| [web #83](https://github.com/MeshCore-Beacon/beacon-web/pull/83) | `0787e5b5fdb1d8d47aefcbb20e816ba280a2ebe2` | `2f76286ec89c81db0345937b8c1712f363bd2e43` | +| [web #85](https://github.com/MeshCore-Beacon/beacon-web/pull/85) | `b21b8704c7403d7f963dc445de7137b9857e194e` | `0787e5b5fdb1d8d47aefcbb20e816ba280a2ebe2` | +| [web #87](https://github.com/MeshCore-Beacon/beacon-web/pull/87) | `be5434a3d4784e26434b5395693896dc7139fd4d` | `b21b8704c7403d7f963dc445de7137b9857e194e` | +| [web #89](https://github.com/MeshCore-Beacon/beacon-web/pull/89) | `037858b4646349c4599d9067fb8f42ad9d1a34a3` | `be5434a3d4784e26434b5395693896dc7139fd4d` | +| [web #92](https://github.com/MeshCore-Beacon/beacon-web/pull/92) | `6b6884951a3dac01b592dfec83f0191879c5696c` | `037858b4646349c4599d9067fb8f42ad9d1a34a3` | +| [web #95](https://github.com/MeshCore-Beacon/beacon-web/pull/95) | `e1133ab5c3364349af900fd4f2093384fbf34f56` | `6b6884951a3dac01b592dfec83f0191879c5696c` | + +## Recovery + +The previous pair is server `2ed2e031` / web `6b688495`. The verified checkpoint and corresponding source/binaries remain, with a checksum-verified private copy off the Pi. Backend rollback restores the broader route index for the old query planner and the previous app/frontend metadata while preserving new rows and the compatible archive constraints. Frontend-only rollback restores `6b688495` while retaining backend `a35cba1d`. Exact guarded operations are kept with the deployment's local evidence; do not apply an earlier phase's rollback command directly to this candidate. From c3eea149f135effa31344907a8d4ab5cc8d8b64b Mon Sep 17 00:00:00 2001 From: n30nex Date: Tue, 29 Sep 2026 20:56:13 -0400 Subject: [PATCH 36/69] docs: record My Atlas PR and verified preview --- CONTRIBUTOR_WORKFLOW.md | 2 ++ RELEASE-CHECKLIST.md | 18 ++++++++++- ROADMAP.md | 20 ++++++++++-- app_documentation/my-atlas-20260929.md | 44 ++++++++++++++++++++++++++ 4 files changed, 81 insertions(+), 3 deletions(-) create mode 100644 app_documentation/my-atlas-20260929.md diff --git a/CONTRIBUTOR_WORKFLOW.md b/CONTRIBUTOR_WORKFLOW.md index 3b2ce0a..d48751b 100644 --- a/CONTRIBUTOR_WORKFLOW.md +++ b/CONTRIBUTOR_WORKFLOW.md @@ -16,6 +16,8 @@ The helper remains useful for the existing queue's integration with new `dev` ch The normal long-term path is a short queue: finish the current review sequence, then start each independent change from fresh `dev`. Keep only real dependencies stacked. Retain the exact built source and existing rollback; a documentation or history-only change with the same source tree needs no application rebuild. +The My Atlas phase is one feature PR, web #97 after #95. It reused the published tip without rebasing earlier parents; source, checks and recovery are recorded in [the My Atlas handoff](app_documentation/my-atlas-20260929.md). + ## Working loop 1. Refresh issues, review feedback and the [roadmap](ROADMAP.md). Choose one logical API, page or correction. diff --git a/RELEASE-CHECKLIST.md b/RELEASE-CHECKLIST.md index f33858b..43dc577 100644 --- a/RELEASE-CHECKLIST.md +++ b/RELEASE-CHECKLIST.md @@ -2,7 +2,23 @@ Status: development preview and review handoff, 29 September 2026 (Toronto). Stable tags and production cutover remain owner-managed. This is not a complete CoreScope parity claim. -## Current integration gate — 29 September +## My Atlas delivered for review — 29 September + +[Web #97](https://github.com/MeshCore-Beacon/beacon-web/pull/97), `4fd4b0de`, is the single My Atlas feature PR requested by the contributor. It follows #95 at `e1133ab5` and closes [web #96](https://github.com/MeshCore-Beacon/beacon-web/issues/96) on acceptance. Earlier application PRs remain included; no parent was rebased for this feature. + +My Atlas sits in the desktop tab row and the phone More menu. Visitors save up to twelve full-key node identities, order and a 24h/3d window in this browser. Compact cards show reception bars, SNR/RSSI meters and server freshness; Heard by and statistics expand on demand. Search collapses on return visits. Node, observer/dashboard and exact packet/path investigation reuse the existing navigation. English and French ship together. + +Counts are explicitly the latest **200 retained origin-key reports per node**, filtered to the selected period. Companion requests and other identified-origin packets are included as well as adverts. This is not a complete node-traffic total. Heard by describes the latest loaded packet, not lifetime reach. Missing readings, expired details and incomplete samples remain visible; no radio-health or packet-loss score is invented. + +The Pi now runs unchanged server `a35cba1d` with web `4fd4b0de`. Windows and native Pi build/lint/all **960 tests** pass, along with the actual published-head CI (web CodeQL skipped). Desktop, French 390px phone, keyboard, persistence/order/removal, packet/observer links and dashboard Back checks pass. The public 19 assets and both source archives match, both MQTT feeds are connected, and all 24 container identities/restart counts are unchanged. Physical iPhone Safari and full production capacity remain separate release gates. + +[My Atlas preview](https://canadaverse.org/beacon-dev/?tab=MyAtlas) · [Changelog and source](https://canadaverse.org/beacon-dev/source.html). Frontend rollback restores `e1133ab5` with server `a35cba1d`, from `web-20260930T004937Z`. For an older backend rollback, restore this frontend first, then use the existing September 29 backend recipe; its guard intentionally rejects an unknown newer frontend. + +The contributor explicitly prioritized My Atlas for this phase. Next: refresh maintainer feedback and issues, then server #183 before optional MeshMapper boundaries. Broader issues and remaining parity work stay open. All eighteen application candidates are out of draft; maintainers retain acceptance, merges, stable releases and production cutover. + +See [My Atlas validation and recovery](app_documentation/my-atlas-20260929.md). + +## Earlier integration gate — 29 September All six server reviews are addressed in their existing PR sequence. The fixes restore analytics indexes, consolidate unmerged migrations, preserve current partial activity buckets, align cache windows, narrow the route index, keep manual scope priority and simplify channel insertion metadata. Independent server #184 fixes RFC3339 offsets; new web #95 follows #92 and matches time choices to retained data. Existing candidates remain included. diff --git a/ROADMAP.md b/ROADMAP.md index ac9da39..2828609 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -6,9 +6,25 @@ Refresh GitHub issues, PR feedback and branch state before starting a phase. Thi Release-check correction, 21 September UTC: the workflow now includes independent preview PRs in Status and Check, applies the same CI/head/fork/target requirements to them, and rechecks the prepared independent inputs before publication. This covers packet summaries #161 and map correction #61 without adding them to the ordered stacks. The backup CLI #160 retains its separate check. Twenty-three offline regressions cover these gates and the existing no-rebase/cache behavior. See [the contributor workflow](CONTRIBUTOR_WORKFLOW.md). -Accepted and integrated bases remain server `db30c9b5` / web `17f48fb9`. The current tested preview is server `a35cba1d` / web `e1133ab5`, with all seventeen application review candidates. The sections below retain dated historical checkpoints; use the September 29 record for current heads and rollback. +Accepted and integrated bases remain server `db30c9b5` / web `17f48fb9`. The current tested preview is server `a35cba1d` / web `4fd4b0de`, with all eighteen application review candidates. The sections below retain dated historical checkpoints; use the My Atlas September 29 record for current heads and rollback. -## Review corrections and history windows — 29 September +## My Atlas delivered for review — 29 September + +[Exact feature, validation and recovery](app_documentation/my-atlas-20260929.md). + +[Web #97](https://github.com/MeshCore-Beacon/beacon-web/pull/97), `4fd4b0de`, is the single My Atlas feature PR requested by the contributor. It follows #95 at `e1133ab5` and closes [web #96](https://github.com/MeshCore-Beacon/beacon-web/issues/96) on acceptance. Earlier application PRs remain included; no parent was rebased for this feature. + +My Atlas sits in the desktop tab row and the phone More menu. Visitors save up to twelve full-key node identities, order and a 24h/3d window in this browser. Compact cards show reception bars, SNR/RSSI meters and server freshness; Heard by and statistics expand on demand. Search collapses on return visits. Node, observer/dashboard and exact packet/path investigation reuse the existing navigation. English and French ship together. + +Counts are explicitly the latest **200 retained origin-key reports per node**, filtered to the selected period. Companion requests and other identified-origin packets are included as well as adverts. This is not a complete node-traffic total. Heard by describes the latest loaded packet, not lifetime reach. Missing readings, expired details and incomplete samples remain visible; no radio-health or packet-loss score is invented. + +The Pi now runs unchanged server `a35cba1d` with web `4fd4b0de`. Windows and native Pi build/lint/all **960 tests** pass, along with the actual published-head CI (web CodeQL skipped). Desktop, French 390px phone, keyboard, persistence/order/removal, packet/observer links and dashboard Back checks pass. The public 19 assets and both source archives match, both MQTT feeds are connected, and all 24 container identities/restart counts are unchanged. Physical iPhone Safari and full production capacity remain separate release gates. + +[My Atlas preview](https://canadaverse.org/beacon-dev/?tab=MyAtlas) · [Changelog and source](https://canadaverse.org/beacon-dev/source.html). Frontend rollback restores `e1133ab5` with server `a35cba1d`, from `web-20260930T004937Z`. For an older backend rollback, restore this frontend first, then use the existing September 29 backend recipe; its guard intentionally rejects an unknown newer frontend. + +The contributor explicitly prioritized My Atlas for this phase. Next: refresh maintainer feedback and issues, then server #183 before optional MeshMapper boundaries. Broader issues and remaining parity work stay open. All eighteen application candidates are out of draft; maintainers retain acceptance, merges, stable releases and production cutover. + +## Earlier review corrections and history windows — 29 September All six server reviews are addressed in their existing PR sequence. The fixes restore analytics indexes, consolidate unmerged migrations, preserve current partial activity buckets, align cache windows, narrow the route index, keep manual scope priority and simplify channel insertion metadata. Independent server #184 fixes RFC3339 offsets; new web #95 follows #92 and matches time choices to retained data. Existing candidates remain included. diff --git a/app_documentation/my-atlas-20260929.md b/app_documentation/my-atlas-20260929.md new file mode 100644 index 0000000..cf10471 --- /dev/null +++ b/app_documentation/my-atlas-20260929.md @@ -0,0 +1,44 @@ +# My Atlas validation and preview handoff — 29 September 2026 + +Feature PR: [beacon-web #97](https://github.com/MeshCore-Beacon/beacon-web/pull/97), closes #96 after acceptance. One frontend feature PR, following #95. Shared documentation continues in the existing beacon-docs #5. + +| Boundary | Exact value | +|---|---| +| Parent | `e1133ab5c3364349af900fd4f2093384fbf34f56` | +| Feature / deployed web | `4fd4b0de5e432527baff4c5f0bc08a8592ec3086` | +| Web tree | `c3bd339f6e730da9b693d4d2278cd85afb20df65` | +| Unchanged server | `a35cba1d2c5cd1dccfa4a99b41717747ebd2f2fc` | +| Web corresponding source SHA-256 | `5a8abb0312aa3a4164b6e06fb1fe38088bae70d3fe8a9978274fe408c7f94fef` | +| Distribution SHA-256 | `e5d9be58d1326884fdb9fa526e47d2afa06b1a1e6dc3ab854c108052b470ff46` | +| Public index SHA-256 | `8d0cacdeba3e63666d8b151a50d4e7edfc1cc4fabe70d4f8ddec34013ea64937` | +| Previous frontend / rollback | `e1133ab5`, backup `web-20260930T004937Z` | + +## Scope and counting + +Saved nodes are browser-local full identities, capped at twelve. Selection, order and 24h/3d window survive reload; clearing site data removes them. Names come from the current record, and an old server ID is recovered only through an exact full public-key match. No account or cross-device sharing is implied by the MyAtlas URL. + +Each card loads up to 200 latest retained origin-key reports and filters that sample to the window. All payload types with a known full origin key count, including companion requests. This is not every packet sent or forwarded by the node. Each bar spans one rolling hour; empty bars mean no loaded reports. The signal meters average finite available last-hop measurements, excluding the zero-RSSI/zero-SNR unavailable sentinel. Numeric equivalents and hourly values are accessible. Freshness uses the server verdict. + +Only the expanded Heard by card fetches its latest loaded packet. Duplicate observer identities collapse; the bounded list opens existing observer inspection/dashboard and exact packet/path investigation. An expired packet is explained. Twelve cards normally cost 24 requests per minute while visible, plus debounced bounded search and one on-demand packet request. Errors stop automatic polling; request cancellation is wired. A changed database ID can add two exact-key recovery requests. No new chart dependency, server code, schema, ingest or retention change. + +## Verification + +- Windows Node 22.18.0 and native Pi Node 24.15.0: build, lint and all 960 tests (109 files) pass. The existing packet virtualizer lint warning remains; no new lint errors. +- Exact published-head [GitHub CI](https://github.com/MeshCore-Beacon/beacon-web/actions/runs/36651632787) passes; CodeQL is skipped under the repository's current workflow. The stack Check covers all current application heads; readiness does not resolve outstanding human change requests on older server PRs. +- Browser checks use real Pi data: desktop and French 390px phone, keyboard controls, saved selection/order/removal/range across reload, mobile More navigation, packet and observer inspection, and dashboard Back to the retained Atlas context. No horizontal phone overflow. Missing/expired data, invalid searches, unavailable storage, cross-tab events, cancellation and identity mismatch are covered by regressions. +- The public preview matches all 19 asset digests and both corresponding-source archives. The changelog lists #97 and preserves earlier candidates. Both MQTT brokers are connected; all 24 existing containers retain their IDs, processes, images and restart counts. Config SHA remains `c09c3aea14eef795baf84d0462dd53c59de5f6e432a6023f93592f5adf6cc277`, mode 0644. Public channel/scopes, raw 72h / analytics 30d / telemetry 720h and disabled public admin/backup/foreign detection are unchanged. +- The established guard assumes origin owns the feature branch. The fork-aware stack workflow runs the local guard and separately verifies the contribution fork SHA, open upstream PR and actual CI; no remotes or protection rules were changed. + +Physical Safari and production-volume browser/network behavior are not established by desktop browser emulation. These cards intentionally sample retained reports; complete per-node totals would need a separately reviewed summary API. This release does not claim complete CoreScope parity. + +## Recovery + +The frontend-only deployment restarted no service and made no database changes. On the Pi, use the phase's guarded helper: + +```sh +python3 /opt/canadaverse/beacon-dev-20260906/evidence/my-atlas-20260929/deploy-beacon-web.py rollback --evidence-dir my-atlas-20260929 +``` + +This restores the prior frontend, source/changelog references and build metadata from `backups/web-20260930T004937Z`, paired with unchanged server `a35cba1d`. Current/rollback assets remain available. If the older backend also needs restoring, first perform this frontend rollback, then use `evidence/review-release-20260929/deploy-server.py rollback`; it restores server `2ed2e031` / web `6b688495` and the old broader route index while preserving new rows and compatible constraints. Do not skip the frontend step or bypass its revision guard. The prior restore-tested private database dump remains separate. + +Maintainers own merges and production release. Next work returns to fresh reviews and server #183, followed by optional boundaries and the remaining parity roadmap. From 81613c32ef57648951019277c3c53e6d70ab6155 Mon Sep 17 00:00:00 2001 From: n30nex Date: Wed, 30 Sep 2026 05:06:20 -0400 Subject: [PATCH 37/69] docs: prepare 1.3.2 release scope and boundary installation --- .github/workflows/contributor-tools.yml | 2 +- CONTRIBUTOR_WORKFLOW.md | 6 + RELEASE-CHECKLIST.md | 6 + ROADMAP.md | 6 + .../meshmapper-border-snapshots.md | 27 +++++ .../meshmapper-boundaries-plan.md | 4 +- app_documentation/release-132-preparation.md | 53 +++++++++ app_documentation/saved-route-evidence.md | 11 ++ tools/download_meshmapper_borders.py | 112 ++++++++++++++++++ tools/test_meshmapper_borders.py | 28 +++++ 10 files changed, 252 insertions(+), 3 deletions(-) create mode 100644 app_documentation/meshmapper-border-snapshots.md create mode 100644 app_documentation/release-132-preparation.md create mode 100644 app_documentation/saved-route-evidence.md create mode 100644 tools/download_meshmapper_borders.py create mode 100644 tools/test_meshmapper_borders.py diff --git a/.github/workflows/contributor-tools.yml b/.github/workflows/contributor-tools.yml index b37fb76..e073e67 100644 --- a/.github/workflows/contributor-tools.yml +++ b/.github/workflows/contributor-tools.yml @@ -17,4 +17,4 @@ jobs: - uses: actions/setup-python@v5 with: python-version: "3.12" - - run: python -m unittest discover -s tools -p test_beacon_stack.py -v + - run: python -m unittest discover -s tools -p "test_*.py" -v diff --git a/CONTRIBUTOR_WORKFLOW.md b/CONTRIBUTOR_WORKFLOW.md index d48751b..453bba8 100644 --- a/CONTRIBUTOR_WORKFLOW.md +++ b/CONTRIBUTOR_WORKFLOW.md @@ -125,3 +125,9 @@ Tests cover squash/drop-parent behavior, a fresh phase after all merges, cache r ## Document encoding Markdown is UTF-8. Read and write it with an explicit UTF-8 encoding in scripts, especially when moving between Windows tools. Check both the diff and rendered text before publication. Keep numeric ranges as en dashes and dependency arrows as arrows; do not round-trip the document through a legacy Windows code page. + +## 1.3.2 release cut + +Keep the active web manifest through #99 and keep Atlas #97 in a separate post-release manifest. Refresh the entire declared dependency sequence after review edits; publish with the recorded remote heads, then check the actual published revisions. The Atlas manifest can include the verified release parents to reuse identical-tree receipts, but must not change the active preview composition. Build and publish the release web archive without Atlas, preserve the prior archive/assets and browser-local saved cards, and keep Atlas based on the release tip for the owner's next phase. + +The bounded border snapshot tool is independent of the stack helper and never deploys or changes configuration. The contributor CI runs all `test_*.py` files, including the stack safety checks and exact-member/null/invalid-polygon coverage (26 tests at this checkpoint). diff --git a/RELEASE-CHECKLIST.md b/RELEASE-CHECKLIST.md index 43dc577..cca4c3c 100644 --- a/RELEASE-CHECKLIST.md +++ b/RELEASE-CHECKLIST.md @@ -1,5 +1,11 @@ # Server/web consolidation release +## 1.3.2 preparation — 30 September + +The current phase prepares web 1.3.2 and holds My Atlas #97 for after release. The release web queue ends at #99; Atlas will remain a separate feature based on that cut. Server review fixes are published and the Pi runs `397d76b3`; the final web review stack and preview are still being validated. Earlier dated preview/rollback descriptions below are historical. + +See [the organisation audit, candidate scope, remaining gates and recovery](app_documentation/release-132-preparation.md), and the [boundary snapshot guide](app_documentation/meshmapper-border-snapshots.md). All four organisation repositories and their open issues/PRs were inspected. Owners retain merges, release tags and production deployment. + Status: development preview and review handoff, 29 September 2026 (Toronto). Stable tags and production cutover remain owner-managed. This is not a complete CoreScope parity claim. ## My Atlas delivered for review — 29 September diff --git a/ROADMAP.md b/ROADMAP.md index 2828609..2cd1ca2 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,5 +1,11 @@ # Beacon parity and analytics roadmap +## 1.3.2 preparation — 30 September + +The current phase prepares web 1.3.2 and holds My Atlas #97 for after release. The release web queue ends at #99; Atlas will remain a separate feature based on that cut. Server review fixes are published and the Pi runs `397d76b3`; the final web review stack and preview are still being validated. Earlier dated preview/rollback descriptions below are historical. + +See [the organisation audit, candidate scope, remaining gates and recovery](app_documentation/release-132-preparation.md), and the [boundary snapshot guide](app_documentation/meshmapper-border-snapshots.md). All four organisation repositories and their open issues/PRs were inspected. Owners retain merges, release tags and production deployment. + Updated 29 September 2026 (Toronto). This is the working roadmap for n30nex's ongoing contributions toward CoreScope feature parity. Maintainers decide acceptance and merge order; deployment owners handle the production switch. Refresh GitHub issues, PR feedback and branch state before starting a phase. This document is a snapshot, and linked issues/PRs are the current source of truth. diff --git a/app_documentation/meshmapper-border-snapshots.md b/app_documentation/meshmapper-border-snapshots.md new file mode 100644 index 0000000..aa18365 --- /dev/null +++ b/app_documentation/meshmapper-border-snapshots.md @@ -0,0 +1,27 @@ +# MeshMapper boundary snapshots + +Beacon can display published MeshMapper polygons through its existing `iatas..borderFile` setting. This release uses reviewed files, with no additional background importer or per-packet network request. + +Run the download tool from this repository, using the public API of the Beacon instance you administer: + +```sh +python3 tools/download_meshmapper_borders.py --beacon-api https://dev.meshcore.ca/api/v1 --output borders-20260930 +``` + +The output must be a new directory. The tool checks the CA and US [Zones API](https://wiki.meshmapper.net/zones-api/) catalogues against Beacon's known IATAs, downloads the corresponding region GeoJSON, and keeps only the exact `properties.code` member. Group collections are not combined; null geometry does not become a radius circle. Downloads are limited to 5 MiB and polygons to 100,000 vertices. The manifest records source URLs, ETags, file hashes and per-region failures. Inspect every error before installation. This command does not change Beacon configuration, the database or services. + +Preserve existing manual borders. Review the downloaded shape and source, then configure only missing boundaries, for example: + +```yaml +iatas: + YKF: + borderFile: borders-20260930/YKF.geojson +``` + +Keep any existing name, latitude and longitude in this entry. Relative paths resolve against Beacon's configuration directory. Containers need a read-only bind mount that makes these files available at that path. Beacon validates Polygon/MultiPolygon features at startup; stage the files with the exact candidate before replacing a working service. Invalid geometry must not replace the last working files. Border changes require a restart. Do not enable foreign-repeater classification merely to draw map boundaries. + +Before deployment, retain the old configuration, runner/mounts and affected IATA metadata as well as a verified database backup. Configuration is imported into IATA records at startup: restoring files alone may leave newly imported metadata in the database. A rollback must restore the previous border/name/centre fields for affected IATAs without deleting new traffic rows. Invalidate the affected IATA/border cache entries or allow their expiry. Verify each `/api/v1/iatas/{iata}/border` response and the map layer after restart. + +Refresh snapshots deliberately; respect the provider's minimum one-hour polling period and retain the previous snapshot. This tool is a one-shot download, not an automatic conditional-refresh service. Automatic refresh with last-known-good handling remains a later feature. + +On 30 September the development Pi accepted 26 exact region polygons, about 1 MiB before Beacon normalization. Both country catalogues were considered; the installed matches were Canadian IATAs. Existing manual polygons were preserved. This is not a claim that every US region has a boundary, nor that a named scope proves a geographic crossing. Scope-name import remains independently configured. diff --git a/app_documentation/meshmapper-boundaries-plan.md b/app_documentation/meshmapper-boundaries-plan.md index 61ef3c1..d05e45c 100644 --- a/app_documentation/meshmapper-boundaries-plan.md +++ b/app_documentation/meshmapper-boundaries-plan.md @@ -1,10 +1,10 @@ # Optional MeshMapper boundary synchronization -Queued after the channel-scope slice, from the maintainer discussion supplied on 27 September. The [published Zones API](https://wiki.meshmapper.net/zones-api/) already supplies the required catalogue and polygons; no scraping or new endpoint is needed. +A reviewed snapshot workflow is available for the 1.3.2 preparation; see [installation and recovery](meshmapper-border-snapshots.md). Automatic synchronization remains a later phase from the maintainer discussion supplied on 27 September. The [published Zones API](https://wiki.meshmapper.net/zones-api/) already supplies the required catalogue and polygons; no scraping or new endpoint is needed. Verified public reads: `https://meshmapper.net/get_zones.php?country=CA` lists enabled regions, and `https://yow.meshmapper.net/get_geojson.php` returns YOW's FeatureCollection. Both are unauthenticated and support ETag caching with a one-hour minimum polling period. Coordinates use longitude, latitude. A missing boundary is explicitly null; group collections contain separate member features. Region codes can be 2–6 alphanumeric characters, so check compatibility with Beacon identifiers before importing or mapping them. Never truncate codes. -Implement an independently enabled boundary source, preserving manual `border_file` precedence. Reuse Beacon's existing GeoJSON validator, border storage and map layer. Bound downloads and vertices, validate the configured region against each feature, preserve source/freshness information and last-known-good geometry, and respect conditional responses and retry timing. Removing a remote boundary must not erase a manual one or synthesize a radius circle. Public foreign-node classification remains disabled unless the owner changes that setting. +Implement an independently enabled boundary source, preserving manual `borderFile` precedence. Reuse Beacon's existing GeoJSON validator, border storage and map layer. Bound downloads and vertices, validate the configured region against each feature, preserve source/freshness information and last-known-good geometry, and respect conditional responses and retry timing. Removing a remote boundary must not erase a manual one or synthesize a radius circle. Public foreign-node classification remains disabled unless the owner changes that setting. Deliver boundary import/map display first; put packet/route crossing investigation in a later focused PR. Crossing views must distinguish observed positions from inferred paths and explain gaps, ambiguous hashes and stale locations. Relate scopes through recorded packet evidence, never through a scope name's spelling. The maintainer explicitly noted that `#ottawa` came from an individual discovered node; that name is not an authoritative city boundary or network-wide forwarding claim. diff --git a/app_documentation/release-132-preparation.md b/app_documentation/release-132-preparation.md new file mode 100644 index 0000000..b2279f1 --- /dev/null +++ b/app_documentation/release-132-preparation.md @@ -0,0 +1,53 @@ +# Beacon 1.3.2 preparation + +30 September 2026. Maintainers choose acceptance, the release commit, tags and the production switch. **1.3.2 is the web version**; the server currently has a v1.6.0 release and needs its own version decision. + +The release composition stops at [web #99](https://github.com/MeshCore-Beacon/beacon-web/pull/99). [My Atlas #97](https://github.com/MeshCore-Beacon/beacon-web/pull/97) is held for after 1.3.2, as one feature PR based on the release work. Removing Atlas from this preview does not clear saved browser cards. + +## Candidate and recovery + +Server candidate `397d76b33488da0c5913356578aa708a44d0e18a` includes the refreshed #167 → #169 → #172 → #174 → #176 sequence, plus independent #166 and #184. Accepted dev remains `db30c9b5`; accepted web dev remains `17f48fb9`. The exact web candidate and final verification receipt will be recorded when the stack checks finish. + +The [development Pi](https://canadaverse.org/beacon-dev/) serves the composed candidate and [corresponding source/changelog](https://canadaverse.org/beacon-dev/source.html). It is a review deployment, not a production release. The owner-managed [dev.meshcore.ca](https://dev.meshcore.ca/) remains a separate deployment; its exact running server commit is not exposed by the public API. + +The server/config rollback checkpoint is `release132-cutover-20260930T021238Z`. Its guarded `evidence/release-132-20260929/deploy-server.py rollback` restores server `a35cba1d`, the prior configuration/runner and affected IATA metadata, retaining new traffic rows. The verified dump was restored into an isolated database and checksum-verified off the Pi. Use the paired frontend rollback before restoring an older server checkpoint; never relabel an existing binary with a rebased commit. + +## User-visible cleanup + +The observer list stays on the left on desktop. On phones, Back opens the retained list. The duplicate dashboard picker, traffic badge, explanatory paragraphs and exact-values disclosure are removed. Status uses a coloured icon with an accessible name. The six metrics and activity charts remain first, with 24h/3d choices and comparison available in context. + +Channels no longer has the scope explanation disclosure. The Analytics retention banner is removed and the Mesh scope list starts collapsed. Desktop navigation starts with Observers; Atlas will follow it after release. English and French changes ship together. + +The missing map layer was a configuration issue: the Pi had no polygons for the affected IATAs. A bounded, reviewed snapshot now fills 26 missing boundaries through the existing border setting. Existing manual borders are preserved. Web caching now retries a missing border and notices a changed shape. See [boundary installation and recovery](meshmapper-border-snapshots.md). The public dev site still returned 204 for YKF/YOW in this audit and needs the same operator configuration step. + +Review corrections also preserve cached charts during transient errors, anchor comparison at the time it is opened/refreshed, avoid polling fixed comparison windows, close competing observer/packet dialogs, preserve selected-report links, clear stale route links on navigation, validate future route windows, and keep the newest channel history page when loading older history. + +## Every repository and open item + +All four organisation repositories were inspected: server, web, docs and mobile. Mobile has no open issues/PRs or recent CI in the inspected metadata; its private contents are not reproduced here. Docs #5 carries the shared release record, operator guides and contributor tooling. No unrelated mobile release is implied by the web version. + +| Repository | Review candidates and linked issues | Disposition | +|---|---|---| +| Server | #166 → #164; #167 → #165; #169 → #168; #172 → #171; #174 → #173; #176 → #175; #184 → #181 | Included. Close focused issues only on acceptance. | +| Web | #75 → #74; #79 → #76; #80 → #77; #81 → #78; #83 → #82; #85 → #84; #87 → #86; #89 → #88; #92 → #90; #95 → #94; #99 → #98 | Release sequence, in this order. Review each focused parent-to-head diff. | +| Web | #97 → #96 | Post-1.3.2 feature; exclude from release artifacts. | +| Server | #183, saved-route prefix metadata after hash-width changes | Maintainer marked non-blocking. Separate correction; do not widen prefix matching to hide the inconsistency. | +| Server | #60 admin; #72 local/remote backup; #99 packet summaries; #116 MQTT timeout investigation | Partially addressed, still open. No blanket closure. Public admin/backup stay disabled. | +| Web | #12 remaining internationalisation | Still open beyond the translated work in this candidate. | +| Server | External #182, bounded CPU profiling | Source-reviewed and upstream CI green at `639148c6`; not in this deployment. Owner integration and a private-host profiling pilot remain separate. | +| Web | External #93, node View on map | Not in this deployment. At `c0dbb101`, no published check results and the author reported pre-existing test failures. It overlaps the changed navigation; rebase/integration and exact-head tests are required before inclusion. | +| Docs | #5 | Shared roadmap, workflow, recovery and operator documentation. | + +The contributor account has read access to the upstream application repositories. Merge commits are disabled there. Maintainers can enable merge commits to accept this sequence without recreating each parent; the contributor helper never merges upstream PRs. An out-of-draft PR or a green build does not replace maintainer acceptance. Requested re-review remains a release gate. + +## Performance evidence and limits + +Native Pi Go tests used PostgreSQL. The 3,200-input/504-scope replay passed with 100 stored packets, 800 observations, 100 decrypted messages, 800 normal and 2,400 opt-in events, 3,200 acknowledgements and zero fixture drops. This is a bounded replay, not universal losslessness. + +A 40-second read-only sample on 30 September, with an observer dashboard active, found both brokers connected on both sites and newest-packet timestamps advancing. Pi app CPU samples ranged 0.04–2.61% and memory 37.69–38.66 MiB; PostgreSQL CPU 0–2.15%, memory about 399–401 MiB. Sixteen public API reads took 0.105–0.200 seconds. The sampled Beacon log contained no queue-full/overflow, dropping, SQL-error, parser-fallback, reconnect or panic mentions. These are short point samples, not capacity or production CPU guarantees. + +The operator's MeshMapper CPU screenshot cannot be attributed to a single Beacon code path from this Pi sample. If #182 is accepted, run its bounded private profiling on the affected host, compare input/event rate, queue drops, database refresh/cleanup time, process CPU/RSS and request latency over matched busy periods, then decide whether further optimisation is needed. Profiles must stay private; no profiling HTTP endpoint is required. Keep rollback ready before promotion. + +Frontend costs remain explicit: one retained origin view preserves the requested Back/filter/scroll behaviour, but stays mounted and can keep its subscriptions. Removing the unrequested extra observer list avoids another duplicate surface. Channel history retains loaded pages so scrolling does not evict the live page; very long visits can accumulate memory. These are targeted follow-ups if measured usage warrants them, not claims of a fully idle background view or bounded total session memory. + +Raw packets stay 72h, hourly summaries 30d and telemetry 720h. Summary history older than already-expired raw data cannot be reconstructed. Historical counter differences with CoreScope remain unexplained until inputs, observer identity, exact windows and duplicate policies are aligned. Physical iPhone Safari and sustained production-volume testing remain owner release checks. diff --git a/app_documentation/saved-route-evidence.md b/app_documentation/saved-route-evidence.md new file mode 100644 index 0000000..282083b --- /dev/null +++ b/app_documentation/saved-route-evidence.md @@ -0,0 +1,11 @@ +# Saved-route observation evidence + +Known-route responses include `pathKey`, a stable identity within the route's IATA. Use `GET /api/v1/routes/{iata}/{pathKey}/observations` to fetch the **full saved route** and retained report references. Search results can contain a subsegment while sharing the full route's key; the evidence response always describes the complete saved sequence. + +The match requires the complete saved `pathBytes`, `hashSize`, hop count and IATA. A compact digest index narrows candidates, but full bytes are still compared. Other hash widths, TRACE readings/intended routes and unclassified legacy observations are excluded. Matching short prefixes does not confirm historical node identities, forwarding or delivery. The stored route counter can include repeated processing and outlive raw reports; it is not a retained-result total. + +The web interface offers **24h / 3d** and caps raw route/comparison selections at 72 hours to match the preview's packet retention. The API retains its bounded 30-day maximum for deployments with longer retention; it cannot recover expired reports. + +`range` defaults to `24h` and accepts durations up to `720h`, anchored on the server. Alternatively supply both `since` and exclusive `until` in epoch milliseconds, with a maximum 30-day span and no future end. `limit` defaults to 50 and is capped at 200. Follow `nextPageCursor` as `pageCursor`; its route, window and microsecond/ID boundary are pinned. Do not combine it with another range, or change its explicit window. Numeric legacy `cursor` is unsupported. Responses include effective window bounds, `matchAvailable`, an empty `items` array when no matching raw evidence remains, and `hasMore`; no total-count scan or packet/message body is added. Malformed saved path metadata is explicitly unavailable, and missing routes return 404. + +Migration 041 builds the compact observation index concurrently. Keep it as a single statement outside a transaction; the existing runner handles an interrupted or already-built index before recording completion. It does not alter retained rows or expiry configuration. Native PostgreSQL tests cover ties below millisecond precision, cursor scope, different widths/sites, TRACE/unknown exclusions, raw expiry, index retry and custom/generic indexed plans. diff --git a/tools/download_meshmapper_borders.py b/tools/download_meshmapper_borders.py new file mode 100644 index 0000000..3f08aea --- /dev/null +++ b/tools/download_meshmapper_borders.py @@ -0,0 +1,112 @@ +"""Download a reviewed border snapshot; never edit Beacon's config or database. + +Usage: python3 tools/download_meshmapper_borders.py --beacon-api https://dev.meshcore.ca/api/v1 --output borders-20260930 +""" +import argparse +import concurrent.futures +import hashlib +import json +import math +import re +import urllib.parse +import urllib.request +from datetime import datetime, timezone +from pathlib import Path + +MAX_BYTES = 5 * 1024 * 1024 +MAX_VERTICES = 100_000 + + +def read_json(url): + request = urllib.request.Request(url, headers={"User-Agent": "Beacon-boundary-snapshot/1.0"}) + with urllib.request.urlopen(request, timeout=30) as response: + data = response.read(MAX_BYTES + 1) + if len(data) > MAX_BYTES: + raise ValueError("Response exceeds 5 MiB") + return json.loads(data), response.headers.get("ETag") + + +def select_border(document, code): + """Select an exact member, never a group's entire collection or a radius circle.""" + if document.get("type") != "FeatureCollection": + raise ValueError("Expected FeatureCollection") + matches = [f for f in document.get("features", []) if f.get("properties", {}).get("code") == code] + if len(matches) != 1: + raise ValueError("Missing or ambiguous region feature") + feature = matches[0] + geometry = feature.get("geometry") + if geometry is None: + return None + if feature.get("type") != "Feature" or geometry.get("type") not in ("Polygon", "MultiPolygon"): + raise ValueError("Expected polygon geometry") + polygons = [geometry["coordinates"]] if geometry["type"] == "Polygon" else geometry["coordinates"] + vertices = 0 + if not polygons: + raise ValueError("Empty geometry") + for polygon in polygons: + if not polygon: + raise ValueError("Empty polygon") + for ring in polygon: + if len(ring) < 4 or ring[0] != ring[-1]: + raise ValueError("Unclosed polygon ring") + for position in ring: + vertices += 1 + if vertices > MAX_VERTICES or len(position) != 2: + raise ValueError("Invalid or oversized coordinates") + if any(type(value) not in (int, float) or not math.isfinite(value) for value in position): + raise ValueError("Non-finite coordinate") + if not (-180 <= position[0] <= 180 and -90 <= position[1] <= 90): + raise ValueError("Coordinate outside geographic bounds") + return feature + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--beacon-api", required=True) + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--countries", nargs="+", choices=("CA", "US"), default=["CA", "US"]) + args = parser.parse_args() + address = urllib.parse.urlsplit(args.beacon_api) + if address.scheme not in ("http", "https") or address.username or address.password or address.query or address.fragment: + parser.error("Use a public API base URL without credentials, query or fragment") + if args.output.exists(): + parser.error("Output must be a new directory; existing snapshots are preserved") + known, _ = read_json(args.beacon_api.rstrip("/") + "/iatas") + codes = {row["iata"] for row in known} + zones = {} + for country in dict.fromkeys(args.countries): + catalogue, _ = read_json("https://meshmapper.net/get_zones.php?country=" + country) + for zone in catalogue["zones"]: + code = zone["code"] + if code in codes and re.fullmatch(r"[A-Z0-9]{2,6}", code) and zone.get("country") == country and zone.get("has_boundary"): + zones[code] = zone + args.output.mkdir(parents=True) + + def download(item): + code, zone = item + # Catalogue-controlled URLs are not arbitrary download destinations. + address = urllib.parse.urlsplit(zone["url"]) + if address.scheme != "https" or address.hostname != code.lower() + ".meshmapper.net" or address.username or address.password or address.port not in (None, 443): + return {"code": code, "error": "Unexpected region URL"} + url = "https://" + address.hostname + "/get_geojson.php" + try: + document, etag = read_json(url) + feature = select_border(document, code) + if feature is None: + return {"code": code, "source": url, "status": "no_boundary"} + raw = (json.dumps(feature, ensure_ascii=False, separators=(",", ":")) + "\n").encode("utf-8") + filename = code + ".geojson" + (args.output / filename).write_bytes(raw) + return {"code": code, "source": url, "etag": etag, "file": filename, "sha256": hashlib.sha256(raw).hexdigest(), "bytes": len(raw)} + except (OSError, ValueError, KeyError, TypeError) as error: + return {"code": code, "source": url, "error": str(error)} + + with concurrent.futures.ThreadPoolExecutor(max_workers=4) as pool: + entries = list(pool.map(download, sorted(zones.items()))) + manifest = {"generated_at": datetime.now(timezone.utc).isoformat(), "beacon_api": args.beacon_api, "countries": args.countries, "entries": entries} + (args.output / "manifest.json").write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8") + print(json.dumps({"downloaded": sum("file" in row for row in entries), "errors": sum("error" in row for row in entries), "manifest": str(args.output / "manifest.json")})) + + +if __name__ == "__main__": + main() diff --git a/tools/test_meshmapper_borders.py b/tools/test_meshmapper_borders.py new file mode 100644 index 0000000..3aff86a --- /dev/null +++ b/tools/test_meshmapper_borders.py @@ -0,0 +1,28 @@ +import unittest +from download_meshmapper_borders import select_border + + +def feature(code="YKF"): + return {"type": "Feature", "properties": {"code": code}, "geometry": {"type": "Polygon", "coordinates": [[[0, 0], [1, 0], [1, 1], [0, 0]]]}} + + +class BorderSelectionTest(unittest.TestCase): + def test_exact_group_member_only(self): + wanted = feature() + self.assertEqual(select_border({"type": "FeatureCollection", "features": [feature("YOW"), wanted]}, "YKF"), wanted) + with self.assertRaises(ValueError): + select_border({"type": "FeatureCollection", "features": [wanted, wanted]}, "YKF") + + def test_null_is_not_a_synthetic_circle(self): + value = feature(); value["geometry"] = None + self.assertIsNone(select_border({"type": "FeatureCollection", "features": [value]}, "YKF")) + + def test_malformed_geometry_fails(self): + for position in ([181, 0], [0, 91], [float("nan"), 0], [False, 0]): + value = feature(); value["geometry"]["coordinates"][0][1] = position + with self.assertRaises(ValueError): + select_border({"type": "FeatureCollection", "features": [value]}, "YKF") + + +if __name__ == "__main__": + unittest.main() From 7e04d2e43399ddc28a1d0008454104c84e74f5e4 Mon Sep 17 00:00:00 2001 From: n30nex Date: Wed, 30 Sep 2026 07:58:48 -0400 Subject: [PATCH 38/69] docs: record verified 1.3.2 cut and accepted server batch --- CONTRIBUTOR_WORKFLOW.md | 2 + RELEASE-CHECKLIST.md | 22 ++- ROADMAP.md | 12 +- app_documentation/my-atlas-20260929.md | 2 + app_documentation/release-132-heads.json | 150 +++++++++++++++++++ app_documentation/release-132-preparation.md | 22 ++- tools/beacon_stack.py | 4 +- 7 files changed, 195 insertions(+), 19 deletions(-) create mode 100644 app_documentation/release-132-heads.json diff --git a/CONTRIBUTOR_WORKFLOW.md b/CONTRIBUTOR_WORKFLOW.md index 453bba8..9e9692d 100644 --- a/CONTRIBUTOR_WORKFLOW.md +++ b/CONTRIBUTOR_WORKFLOW.md @@ -131,3 +131,5 @@ Markdown is UTF-8. Read and write it with an explicit UTF-8 encoding in scripts, Keep the active web manifest through #99 and keep Atlas #97 in a separate post-release manifest. Refresh the entire declared dependency sequence after review edits; publish with the recorded remote heads, then check the actual published revisions. The Atlas manifest can include the verified release parents to reuse identical-tree receipts, but must not change the active preview composition. Build and publish the release web archive without Atlas, preserve the prior archive/assets and browser-local saved cards, and keep Atlas based on the release tip for the owner's next phase. The bounded border snapshot tool is independent of the stack helper and never deploys or changes configuration. The contributor CI runs all `test_*.py` files, including the stack safety checks and exact-member/null/invalid-polygon coverage (26 tests at this checkpoint). + +The rebuild comparison now uses `preview.web_tree` for web source and `preview.server_tree` for server source, retaining the older field as a fallback. This prevents a web history-only refresh from comparing against the backend tree and asking for an unnecessary Pi rebuild. The actual deployed revision and source archive stay unchanged when the source tree matches. The release web refresh was verified to report no rebuild. diff --git a/RELEASE-CHECKLIST.md b/RELEASE-CHECKLIST.md index cca4c3c..075a0c8 100644 --- a/RELEASE-CHECKLIST.md +++ b/RELEASE-CHECKLIST.md @@ -1,14 +1,26 @@ # Server/web consolidation release -## 1.3.2 preparation — 30 September +## 1.3.2 release preparation — 30 September -The current phase prepares web 1.3.2 and holds My Atlas #97 for after release. The release web queue ends at #99; Atlas will remain a separate feature based on that cut. Server review fixes are published and the Pi runs `397d76b3`; the final web review stack and preview are still being validated. Earlier dated preview/rollback descriptions below are historical. +The Pi now serves server `397d76b3` / web `eb241ca7`, with Atlas excluded. All seven server release PRs are accepted in dev `89a376c2`, whose source tree exactly matches the running server; no rebuild or relabel was needed. Web #75 → #79 → #80 → #81 → #83 → #85 → #87 → #89 → #92 → #95 → #99 is published and passes checks. The native/Windows release build passes all 955 tests, and all 21 public assets and source archives match. -See [the organisation audit, candidate scope, remaining gates and recovery](app_documentation/release-132-preparation.md), and the [boundary snapshot guide](app_documentation/meshmapper-border-snapshots.md). All four organisation repositories and their open issues/PRs were inspected. Owners retain merges, release tags and production deployment. +My Atlas #97 is held for after 1.3.2 at `66ae0cc2`, directly after #99. Its requested copy/order changes, Windows/Pi 975 tests, CI and browser checks pass; it is not deployed. External server #182 and web #93 remain separate gates. -Status: development preview and review handoff, 29 September 2026 (Toronto). Stable tags and production cutover remain owner-managed. This is not a complete CoreScope parity claim. +[Current audit, exact heads, validation and recovery](app_documentation/release-132-preparation.md). [Live candidate and source](https://canadaverse.org/beacon-dev/source.html). Maintainers retain web acceptance, tags, version decisions and production rollout. Earlier dated records below are historical. -## My Atlas delivered for review — 29 September +## Owner release gates for 1.3.2 + +- [x] Accept the seven server release PRs. Accepted dev `89a376c2` has passing CI and the same tree as the validated running server. +- [ ] Re-review and accept the eleven release web PRs through #99; check CI on the accepted merge result. +- [ ] Accept docs #5 so the canonical operator links in server #172/#174 resolve. Merge the remaining web sequence in dependency order. +- [ ] Decide separately whether server #182 and web #93 belong in the cut. The current tested composition excludes them; do not silently label an untested combination as this candidate. +- [ ] Confirm the deployed source/binary pair on the affected MeshMapper host, its packet/summary retention, broker inputs, bounded queues, and private rollback. Apply reviewed border files only where manual boundaries are missing. +- [ ] Compare matched busy periods for accepted/dropped inputs, database work, process CPU/RSS and request latency. The Pi replay and short runtime sample do not certify production capacity. If using #182, keep profiles private and bounded. +- [ ] Check the main operator journeys on desktop and physical iPhone Safari, including English/French, Back, dialogs, maps and expired records. +- [ ] Promote the accepted web source to main and cut **web v1.3.2** under the repository's release process. Choose the server version independently of its existing v1.6.0 tag. Publish matching source and retain rollback artifacts. +- [ ] Leave My Atlas #97 out of the release. Keep its one feature branch based on the accepted release work and refresh it after any squash/rebase merge before post-release acceptance. + +## Historical Atlas preview — 29 September [Web #97](https://github.com/MeshCore-Beacon/beacon-web/pull/97), `4fd4b0de`, is the single My Atlas feature PR requested by the contributor. It follows #95 at `e1133ab5` and closes [web #96](https://github.com/MeshCore-Beacon/beacon-web/issues/96) on acceptance. Earlier application PRs remain included; no parent was rebased for this feature. diff --git a/ROADMAP.md b/ROADMAP.md index 2cd1ca2..490d6f5 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,10 +1,12 @@ # Beacon parity and analytics roadmap -## 1.3.2 preparation — 30 September +## 1.3.2 release preparation — 30 September -The current phase prepares web 1.3.2 and holds My Atlas #97 for after release. The release web queue ends at #99; Atlas will remain a separate feature based on that cut. Server review fixes are published and the Pi runs `397d76b3`; the final web review stack and preview are still being validated. Earlier dated preview/rollback descriptions below are historical. +The Pi now serves server `397d76b3` / web `eb241ca7`, with Atlas excluded. All seven server release PRs are accepted in dev `89a376c2`, whose source tree exactly matches the running server; no rebuild or relabel was needed. Web #75 → #79 → #80 → #81 → #83 → #85 → #87 → #89 → #92 → #95 → #99 is published and passes checks. The native/Windows release build passes all 955 tests, and all 21 public assets and source archives match. -See [the organisation audit, candidate scope, remaining gates and recovery](app_documentation/release-132-preparation.md), and the [boundary snapshot guide](app_documentation/meshmapper-border-snapshots.md). All four organisation repositories and their open issues/PRs were inspected. Owners retain merges, release tags and production deployment. +My Atlas #97 is held for after 1.3.2 at `66ae0cc2`, directly after #99. Its requested copy/order changes, Windows/Pi 975 tests, CI and browser checks pass; it is not deployed. External server #182 and web #93 remain separate gates. + +[Current audit, exact heads, validation and recovery](app_documentation/release-132-preparation.md). [Live candidate and source](https://canadaverse.org/beacon-dev/source.html). Maintainers retain web acceptance, tags, version decisions and production rollout. Earlier dated records below are historical. Updated 29 September 2026 (Toronto). This is the working roadmap for n30nex's ongoing contributions toward CoreScope feature parity. Maintainers decide acceptance and merge order; deployment owners handle the production switch. @@ -12,9 +14,9 @@ Refresh GitHub issues, PR feedback and branch state before starting a phase. Thi Release-check correction, 21 September UTC: the workflow now includes independent preview PRs in Status and Check, applies the same CI/head/fork/target requirements to them, and rechecks the prepared independent inputs before publication. This covers packet summaries #161 and map correction #61 without adding them to the ordered stacks. The backup CLI #160 retains its separate check. Twenty-three offline regressions cover these gates and the existing no-rebase/cache behavior. See [the contributor workflow](CONTRIBUTOR_WORKFLOW.md). -Accepted and integrated bases remain server `db30c9b5` / web `17f48fb9`. The current tested preview is server `a35cba1d` / web `4fd4b0de`, with all eighteen application review candidates. The sections below retain dated historical checkpoints; use the My Atlas September 29 record for current heads and rollback. +The following sections preserve dated historical checkpoints. Use the 30 September release record above for current source, acceptance and rollback. -## My Atlas delivered for review — 29 September +## Historical Atlas preview — 29 September [Exact feature, validation and recovery](app_documentation/my-atlas-20260929.md). diff --git a/app_documentation/my-atlas-20260929.md b/app_documentation/my-atlas-20260929.md index cf10471..dd3c5b1 100644 --- a/app_documentation/my-atlas-20260929.md +++ b/app_documentation/my-atlas-20260929.md @@ -1,3 +1,5 @@ +> Historical preview record. Atlas is now held for after web 1.3.2; see [the current release record](release-132-preparation.md). + # My Atlas validation and preview handoff — 29 September 2026 Feature PR: [beacon-web #97](https://github.com/MeshCore-Beacon/beacon-web/pull/97), closes #96 after acceptance. One frontend feature PR, following #95. Shared documentation continues in the existing beacon-docs #5. diff --git a/app_documentation/release-132-heads.json b/app_documentation/release-132-heads.json new file mode 100644 index 0000000..737d0f9 --- /dev/null +++ b/app_documentation/release-132-heads.json @@ -0,0 +1,150 @@ +{ + "verified_at": "2026-09-30T11:44:13.234082+00:00", + "web_version": "1.3.2 candidate", + "server": { + "accepted_dev": "89a376c29198ea0bd006953ff3b319e69d9a4082", + "source_tree": "a7bfdb193f4f6c4c9451ceedbc19531f21413036", + "running_revision": "397d76b33488da0c5913356578aa708a44d0e18a", + "identical_tree": true, + "accepted": [ + { + "pr": 166, + "commit": "97cc22a5a0aa2256b4f8f9abe163bd3bc9c8deca", + "accepted_head": "319df7da5246a71f9693f923d2124bf831eabb82", + "merged_at": "2026-09-30T11:33:46Z", + "url": "https://github.com/MeshCore-Beacon/beacon-server/pull/166" + }, + { + "pr": 167, + "commit": "fd7c6b3eb7b520eeb171e8fc1378846ec1954a05", + "accepted_head": "0c19e95132760eda1bdcf550ae1b01865bbf3c41", + "merged_at": "2026-09-30T11:16:27Z", + "url": "https://github.com/MeshCore-Beacon/beacon-server/pull/167" + }, + { + "pr": 169, + "commit": "b8a8188fbcee30ba456223029044fbd4e59821df", + "accepted_head": "b8a8188fbcee30ba456223029044fbd4e59821df", + "merged_at": "2026-09-30T11:31:11Z", + "url": "https://github.com/MeshCore-Beacon/beacon-server/pull/169" + }, + { + "pr": 172, + "commit": "831148ff5f403fcef9c48943390e420c00b39ae8", + "accepted_head": "831148ff5f403fcef9c48943390e420c00b39ae8", + "merged_at": "2026-09-30T11:31:12Z", + "url": "https://github.com/MeshCore-Beacon/beacon-server/pull/172" + }, + { + "pr": 174, + "commit": "7e01abf5a08156d161257974bb6115189925fbd2", + "accepted_head": "7e01abf5a08156d161257974bb6115189925fbd2", + "merged_at": "2026-09-30T11:31:11Z", + "url": "https://github.com/MeshCore-Beacon/beacon-server/pull/174" + }, + { + "pr": 176, + "commit": "6bf58acb805ffc34bee1a3791e2b1cb76ab24e6a", + "accepted_head": "6bf58acb805ffc34bee1a3791e2b1cb76ab24e6a", + "merged_at": "2026-09-30T11:31:11Z", + "url": "https://github.com/MeshCore-Beacon/beacon-server/pull/176" + }, + { + "pr": 184, + "commit": "89a376c29198ea0bd006953ff3b319e69d9a4082", + "accepted_head": "c39df3043931dbe37d51a73d15f050a1c61d1a3b", + "merged_at": "2026-09-30T11:33:54Z", + "url": "https://github.com/MeshCore-Beacon/beacon-server/pull/184" + } + ], + "verified_at": "2026-09-30T11:39:53.187645+00:00" + }, + "web": { + "accepted_dev": "17f48fb932facbdcc45068afd2db6036ac2ec94d", + "revision": "eb241ca78b320364aa0eaf1c35e41f45a2362023", + "tree": "c58c49c8a87da1dd89dfdf5ce95c6aa1779a8650", + "prs": [ + { + "pr": 75, + "base": "17f48fb932facbdcc45068afd2db6036ac2ec94d", + "head": "f4092a492c506c0b81c8067a62c1f7635607b782" + }, + { + "pr": 79, + "base": "f4092a492c506c0b81c8067a62c1f7635607b782", + "head": "ab5bc0c38963cfb91a64b3fb18c8c2e3d64b4bba" + }, + { + "pr": 80, + "base": "ab5bc0c38963cfb91a64b3fb18c8c2e3d64b4bba", + "head": "368ba1801681afcf77a65d54a4d411c5962e36ba" + }, + { + "pr": 81, + "base": "368ba1801681afcf77a65d54a4d411c5962e36ba", + "head": "4b207fafd28c5637177f44f1b0025072ce90cd6d" + }, + { + "pr": 83, + "base": "4b207fafd28c5637177f44f1b0025072ce90cd6d", + "head": "327ea06e92a1fbbdc2674d14ec8386216ce63a2a" + }, + { + "pr": 85, + "base": "327ea06e92a1fbbdc2674d14ec8386216ce63a2a", + "head": "7a82605e8ecbf343bc5f0d9d4b800a43d7243ab5" + }, + { + "pr": 87, + "base": "7a82605e8ecbf343bc5f0d9d4b800a43d7243ab5", + "head": "c0c11829bc6013ff90294557c3150844a5cfa09c" + }, + { + "pr": 89, + "base": "c0c11829bc6013ff90294557c3150844a5cfa09c", + "head": "19bbbd124982584841e76a4f6e14a61474c87e3a" + }, + { + "pr": 92, + "base": "19bbbd124982584841e76a4f6e14a61474c87e3a", + "head": "fa64057c67fad158072ddf823b0edb0cbc4ad184" + }, + { + "pr": 95, + "base": "fa64057c67fad158072ddf823b0edb0cbc4ad184", + "head": "041a000d9c23c325a65290907fadaa017623132b" + }, + { + "pr": 99, + "base": "041a000d9c23c325a65290907fadaa017623132b", + "head": "eb241ca78b320364aa0eaf1c35e41f45a2362023" + } + ], + "tests": 955, + "source_sha256": "97f21fb63763ea7e7916cfa7caf24a85eb20e6d1206c01b89af4ace97a036720", + "dist_sha256": "0fdc156ba78a9fdf136a3e128cc3767f2ffda1c20e512a0099d3cdd3a2978ce6", + "public_assets": 21 + }, + "post_release_atlas": { + "pr": 97, + "parent": "eb241ca78b320364aa0eaf1c35e41f45a2362023", + "head": "66ae0cc222a6580b52f62051025453a992956cc6", + "tests": 975, + "deployed": false + }, + "excluded_candidates": { + "server": [ + 182 + ], + "web": [ + 93, + 97 + ] + }, + "recovery": { + "server_checkpoint": "release132-cutover-20260930T021238Z", + "web_checkpoint": "web-20260930T110037Z", + "config_sha256": "b3d96f52b25e38a0f3ecfb15cd2e492db1d4f494b18774598a4cfad295e2d62f", + "server_binary_sha256": "7f114816f6abb316cf388bfe0f33cd72c1d15cb7ceee50d66f01c8c9b658b76e" + } +} diff --git a/app_documentation/release-132-preparation.md b/app_documentation/release-132-preparation.md index b2279f1..67edf16 100644 --- a/app_documentation/release-132-preparation.md +++ b/app_documentation/release-132-preparation.md @@ -1,16 +1,20 @@ # Beacon 1.3.2 preparation -30 September 2026. Maintainers choose acceptance, the release commit, tags and the production switch. **1.3.2 is the web version**; the server currently has a v1.6.0 release and needs its own version decision. +30 September 2026. Maintainers choose acceptance, the release commit, tags and the production switch. **1.3.2 is the web version**; the server currently has a v1.6.0 release and needs its own version decision. The dev interface reports 1.3.1, while the latest published web GitHub release remains v1.3.0. The release composition stops at [web #99](https://github.com/MeshCore-Beacon/beacon-web/pull/99). [My Atlas #97](https://github.com/MeshCore-Beacon/beacon-web/pull/97) is held for after 1.3.2, as one feature PR based on the release work. Removing Atlas from this preview does not clear saved browser cards. ## Candidate and recovery -Server candidate `397d76b33488da0c5913356578aa708a44d0e18a` includes the refreshed #167 → #169 → #172 → #174 → #176 sequence, plus independent #166 and #184. Accepted dev remains `db30c9b5`; accepted web dev remains `17f48fb9`. The exact web candidate and final verification receipt will be recorded when the stack checks finish. +Server candidate `397d76b33488da0c5913356578aa708a44d0e18a` includes the refreshed #167 → #169 → #172 → #174 → #176 sequence, plus independent #166 and #184. All seven server PRs are now accepted in `dev` at `89a376c2`. Its `a7bfdb19` tree exactly matches the running `397d76b3` source, so acceptance required no rebuild or restart. Web `eb241ca7` is the current release candidate on accepted web dev `17f48fb9`; it contains all eleven release web PRs through #99. [Exact heads and validation metadata](release-132-heads.json). -The [development Pi](https://canadaverse.org/beacon-dev/) serves the composed candidate and [corresponding source/changelog](https://canadaverse.org/beacon-dev/source.html). It is a review deployment, not a production release. The owner-managed [dev.meshcore.ca](https://dev.meshcore.ca/) remains a separate deployment; its exact running server commit is not exposed by the public API. +The [development Pi](https://canadaverse.org/beacon-dev/) serves server `397d76b3` / web `eb241ca7`, with Atlas excluded. Its [corresponding source/changelog](https://canadaverse.org/beacon-dev/source.html) identifies the review deployment. The owner-managed [dev.meshcore.ca](https://dev.meshcore.ca/) remains a separate deployment. Browser inspection shows web 1.3.1 and its older observer detail pane; its exact running server commit is not exposed by the public API. -The server/config rollback checkpoint is `release132-cutover-20260930T021238Z`. Its guarded `evidence/release-132-20260929/deploy-server.py rollback` restores server `a35cba1d`, the prior configuration/runner and affected IATA metadata, retaining new traffic rows. The verified dump was restored into an isolated database and checksum-verified off the Pi. Use the paired frontend rollback before restoring an older server checkpoint; never relabel an existing binary with a rebased commit. +Windows and native Pi web build/lint and all **955 tests** pass. All eleven current release web heads pass build CI; web CodeQL is skipped by its existing workflow. All **21 public assets** and both source archives match the build, both MQTT inputs are connected, and static frontend publication left all **24 containers unchanged**. Desktop, 390px French phone, keyboard, comparison, shared path links, exact phone-list scroll restoration and dark/light border rendering pass browser checks. A deliberately paused map-module request confirmed that Close/Escape still work during loading. + +The post-release Atlas head is `66ae0cc2`, directly after `eb241ca7`. Windows/Pi build/lint, all **975 tests**, current-head build CI and English/French browser checks pass. The marked storage paragraph is removed; Observers precedes My Atlas in navigation. Existing cards survive reload and Atlas-to-observer Back preserves state. This candidate was built separately and is not deployed. + +The server/config rollback checkpoint is `release132-cutover-20260930T021238Z`. Its guarded `evidence/release-132-20260929/deploy-server.py rollback` restores server `a35cba1d`, the prior configuration/runner and affected IATA metadata, retaining new traffic rows. The verified dump was restored into an isolated database and checksum-verified off the Pi. Both phase-owned test databases were then retired after checking for active sessions; the working dump and recovery files remain. The frontend-only checkpoint `web-20260930T110037Z` restores `4fd4b0de` with server `397d76b3` using `evidence/release-132-20260929/deploy-beacon-web.py rollback --evidence-dir release-132-20260929`. The current server rollback explicitly accepts either the release frontend or that previous frontend and restores the paired older build. Never relabel an existing binary with a rebased commit. ## User-visible cleanup @@ -24,24 +28,26 @@ Review corrections also preserve cached charts during transient errors, anchor c ## Every repository and open item -All four organisation repositories were inspected: server, web, docs and mobile. Mobile has no open issues/PRs or recent CI in the inspected metadata; its private contents are not reproduced here. Docs #5 carries the shared release record, operator guides and contributor tooling. No unrelated mobile release is implied by the web version. +All four organisation repositories were inspected: server, web, docs and mobile. Mobile has no open issues/PRs or recent CI in the inspected metadata; its private contents are not reproduced here. Docs #5 carries the shared release record, operator guides and contributor tooling. The final audit has five open server issues plus profiling PR #182, thirteen web issues and thirteen web PRs, docs PR #5, and no open mobile items. Broader issues remain open; accepted server issues were closed separately. No unrelated mobile release is implied by the web version. | Repository | Review candidates and linked issues | Disposition | |---|---|---| -| Server | #166 → #164; #167 → #165; #169 → #168; #172 → #171; #174 → #173; #176 → #175; #184 → #181 | Included. Close focused issues only on acceptance. | +| Server | #166 → #164; #167 → #165; #169 → #168; #172 → #171; #174 → #173; #176 → #175; #184 → #181 | Accepted; focused issues #164/#165/#168/#171/#173/#175/#181 are closed. | | Web | #75 → #74; #79 → #76; #80 → #77; #81 → #78; #83 → #82; #85 → #84; #87 → #86; #89 → #88; #92 → #90; #95 → #94; #99 → #98 | Release sequence, in this order. Review each focused parent-to-head diff. | | Web | #97 → #96 | Post-1.3.2 feature; exclude from release artifacts. | | Server | #183, saved-route prefix metadata after hash-width changes | Maintainer marked non-blocking. Separate correction; do not widen prefix matching to hide the inconsistency. | | Server | #60 admin; #72 local/remote backup; #99 packet summaries; #116 MQTT timeout investigation | Partially addressed, still open. No blanket closure. Public admin/backup stay disabled. | | Web | #12 remaining internationalisation | Still open beyond the translated work in this candidate. | | Server | External #182, bounded CPU profiling | Source-reviewed and upstream CI green at `639148c6`; not in this deployment. Owner integration and a private-host profiling pilot remain separate. | -| Web | External #93, node View on map | Not in this deployment. At `c0dbb101`, no published check results and the author reported pre-existing test failures. It overlaps the changed navigation; rebase/integration and exact-head tests are required before inclusion. | +| Web | External #93, node View on map | Not in this deployment. At `c0dbb101`, no published check results and the author reported pre-existing test failures. It overlaps the changed navigation, and its new button label is English-only. Integration with the current panel stack, French text and exact-head tests are required before inclusion. | | Docs | #5 | Shared roadmap, workflow, recovery and operator documentation. | -The contributor account has read access to the upstream application repositories. Merge commits are disabled there. Maintainers can enable merge commits to accept this sequence without recreating each parent; the contributor helper never merges upstream PRs. An out-of-draft PR or a green build does not replace maintainer acceptance. Requested re-review remains a release gate. +The contributor account has read access to the upstream application repositories. Merge commits are disabled there. Maintainers can enable merge commits to accept this sequence without recreating each parent; the contributor helper never merges upstream PRs. An out-of-draft PR or a green build does not replace maintainer acceptance. Web re-review remains a release gate. The server batch is accepted and accepted-head CI, coverage, CodeQL and image publication pass. Accept docs #5 so the operator-guide links in the now-merged server #172/#174 resolve on docs main. ## Performance evidence and limits +A controlled frontend build moved the packet-path canvas behind on-demand loading while keeping its dialog controls available. Initial static JavaScript fell from 1,628,804 to 598,019 bytes; gzip estimates fell from 448,100 to 172,032 bytes (about 62%). These totals follow the build manifest's static imports and exclude later on-demand chunks. This measures the initial payload, not a production latency or server-CPU improvement. + Native Pi Go tests used PostgreSQL. The 3,200-input/504-scope replay passed with 100 stored packets, 800 observations, 100 decrypted messages, 800 normal and 2,400 opt-in events, 3,200 acknowledgements and zero fixture drops. This is a bounded replay, not universal losslessness. A 40-second read-only sample on 30 September, with an observer dashboard active, found both brokers connected on both sites and newest-packet timestamps advancing. Pi app CPU samples ranged 0.04–2.61% and memory 37.69–38.66 MiB; PostgreSQL CPU 0–2.15%, memory about 399–401 MiB. Sixteen public API reads took 0.105–0.200 seconds. The sampled Beacon log contained no queue-full/overflow, dropping, SQL-error, parser-fallback, reconnect or panic mentions. These are short point samples, not capacity or production CPU guarantees. diff --git a/tools/beacon_stack.py b/tools/beacon_stack.py index 7771090..533d7b0 100644 --- a/tools/beacon_stack.py +++ b/tools/beacon_stack.py @@ -362,8 +362,10 @@ def main(mode, manifest_path=MANIFEST, state=STATE, branch=None): # Refresh the independent inputs too, after potentially lengthy builds. overlays = active_overlays(manifest) combined_tree = preview_tree(repo, parent, [overlay['head'] for overlay in overlays]) + preview = manifest.get('preview', {}) + deployed_tree = preview.get(kind + '_tree', preview.get('server_tree')) plan = dict(base=base, entries=entries, preview_overlays=overlays, squash_merge_proof=proof, preview_tree=combined_tree, - pi_rebuild_needed=combined_tree != manifest.get('preview', {}).get('server_tree')) + pi_rebuild_needed=combined_tree != deployed_tree) verify_publish_state(manifest, plan) write(state/'prepared.json', plan) print('Stack merge order verified. Pi rebuild needed: '+str(plan['pi_rebuild_needed']), flush=True) From 06ce5ee485617eaa1c078ce25ce3579f09986524 Mon Sep 17 00:00:00 2001 From: n30nex Date: Wed, 30 Sep 2026 11:55:40 -0400 Subject: [PATCH 39/69] docs: record latest merged preview and release follow-up --- RELEASE-CHECKLIST.md | 13 +- ROADMAP.md | 4 +- app_documentation/release-132-heads.json | 230 +++++++++++++++---- app_documentation/release-132-preparation.md | 19 ++ 4 files changed, 217 insertions(+), 49 deletions(-) diff --git a/RELEASE-CHECKLIST.md b/RELEASE-CHECKLIST.md index 075a0c8..623cdae 100644 --- a/RELEASE-CHECKLIST.md +++ b/RELEASE-CHECKLIST.md @@ -2,18 +2,19 @@ ## 1.3.2 release preparation — 30 September -The Pi now serves server `397d76b3` / web `eb241ca7`, with Atlas excluded. All seven server release PRs are accepted in dev `89a376c2`, whose source tree exactly matches the running server; no rebuild or relabel was needed. Web #75 → #79 → #80 → #81 → #83 → #85 → #87 → #89 → #92 → #95 → #99 is published and passes checks. The native/Windows release build passes all 955 tests, and all 21 public assets and source archives match. +The Pi now serves exact merged dev server `689bc232` / web `00d859d9`, with Atlas excluded. The accepted web stack and #100–#103, server #182 and later maintainer fixes are included. Native PostgreSQL tests, the 3,200-input replay, web build/lint/all 1,017 tests and public/browser checks pass. Migration 043 passed on the restored backup without changing raw counts; all 21 assets, source archives and 26 boundaries match. -My Atlas #97 is held for after 1.3.2 at `66ae0cc2`, directly after #99. Its requested copy/order changes, Windows/Pi 975 tests, CI and browser checks pass; it is not deployed. External server #182 and web #93 remain separate gates. +My Atlas #97 is held post-release at `66ae0cc2` and needs a conflict refresh against current dev. Web #93 was closed without merge. The known test-readiness race and first replay timeout are preserved in the validation record; the final serialized replay passed in 13.26 seconds. These checks do not certify sustained production capacity. [Current audit, exact heads, validation and recovery](app_documentation/release-132-preparation.md). [Live candidate and source](https://canadaverse.org/beacon-dev/source.html). Maintainers retain web acceptance, tags, version decisions and production rollout. Earlier dated records below are historical. ## Owner release gates for 1.3.2 -- [x] Accept the seven server release PRs. Accepted dev `89a376c2` has passing CI and the same tree as the validated running server. -- [ ] Re-review and accept the eleven release web PRs through #99; check CI on the accepted merge result. -- [ ] Accept docs #5 so the canonical operator links in server #172/#174 resolve. Merge the remaining web sequence in dependency order. -- [ ] Decide separately whether server #182 and web #93 belong in the cut. The current tested composition excludes them; do not silently label an untested combination as this candidate. +- [x] Accept the server release work and verify the current dev head `689bc232`; CI and native tests pass on the deployed source. +- [x] Accept the release web stack via #99 and follow-ups #100–#103; CI and native tests pass on deployed `00d859d9`. +- [ ] Accept docs #5 so the canonical operator links in server #172/#174 resolve. +- [x] Record the additional dispositions: server #182 is merged and tested, profiling remains disabled on the preview, and web #93 is closed without merge. +- [ ] Resolve the restored 7d/30d UI controls from #101 against the earlier 24h/3d raw-history requirement. Old 3d observer links currently select 7d. - [ ] Confirm the deployed source/binary pair on the affected MeshMapper host, its packet/summary retention, broker inputs, bounded queues, and private rollback. Apply reviewed border files only where manual boundaries are missing. - [ ] Compare matched busy periods for accepted/dropped inputs, database work, process CPU/RSS and request latency. The Pi replay and short runtime sample do not certify production capacity. If using #182, keep profiles private and bounded. - [ ] Check the main operator journeys on desktop and physical iPhone Safari, including English/French, Back, dialogs, maps and expired records. diff --git a/ROADMAP.md b/ROADMAP.md index 490d6f5..da956b0 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -2,9 +2,9 @@ ## 1.3.2 release preparation — 30 September -The Pi now serves server `397d76b3` / web `eb241ca7`, with Atlas excluded. All seven server release PRs are accepted in dev `89a376c2`, whose source tree exactly matches the running server; no rebuild or relabel was needed. Web #75 → #79 → #80 → #81 → #83 → #85 → #87 → #89 → #92 → #95 → #99 is published and passes checks. The native/Windows release build passes all 955 tests, and all 21 public assets and source archives match. +The Pi now serves exact merged dev server `689bc232` / web `00d859d9`, with Atlas excluded. The web stack landed through #99, followed by #100–#103. Server #182 and the maintainer ingestion, caching and location fixes are included. Native PostgreSQL tests, the 3,200-input replay, and web build/lint/all **1,017 tests** pass. All 21 public assets, both source archives and 26 boundaries match. -My Atlas #97 is held for after 1.3.2 at `66ae0cc2`, directly after #99. Its requested copy/order changes, Windows/Pi 975 tests, CI and browser checks pass; it is not deployed. External server #182 and web #93 remain separate gates. +My Atlas #97 remains held at `66ae0cc2` for after 1.3.2 and needs conflict resolution against the new dev branch. Web #93 is closed without merge. Upstream #101 restored 7d/30d controls and maps old 3d observer links to 7d; this conflicts with the requested 24h/3d raw-history controls and remains a release follow-up. Migration 043 was restore-tested: 273 stale location records cleared, raw counts and other located nodes preserved. The prior server/frontend and a verified on/off-Pi dump remain recoverable. [Current audit, exact heads, validation and recovery](app_documentation/release-132-preparation.md). [Live candidate and source](https://canadaverse.org/beacon-dev/source.html). Maintainers retain web acceptance, tags, version decisions and production rollout. Earlier dated records below are historical. diff --git a/app_documentation/release-132-heads.json b/app_documentation/release-132-heads.json index 737d0f9..6545c82 100644 --- a/app_documentation/release-132-heads.json +++ b/app_documentation/release-132-heads.json @@ -1,10 +1,10 @@ { - "verified_at": "2026-09-30T11:44:13.234082+00:00", + "verified_at": "2026-09-30T15:54:13.053817+00:00", "web_version": "1.3.2 candidate", "server": { - "accepted_dev": "89a376c29198ea0bd006953ff3b319e69d9a4082", - "source_tree": "a7bfdb193f4f6c4c9451ceedbc19531f21413036", - "running_revision": "397d76b33488da0c5913356578aa708a44d0e18a", + "accepted_dev": "689bc2326a0462bb387c6087f5cb3514cca636f1", + "source_tree": "a9c3c47b8111b26decffa945d4963952060442f5", + "running_revision": "689bc2326a0462bb387c6087f5cb3514cca636f1", "identical_tree": true, "accepted": [ { @@ -57,72 +57,222 @@ "url": "https://github.com/MeshCore-Beacon/beacon-server/pull/184" } ], - "verified_at": "2026-09-30T11:39:53.187645+00:00" + "verified_at": "2026-09-30T15:54:13.053817+00:00" }, "web": { - "accepted_dev": "17f48fb932facbdcc45068afd2db6036ac2ec94d", - "revision": "eb241ca78b320364aa0eaf1c35e41f45a2362023", - "tree": "c58c49c8a87da1dd89dfdf5ce95c6aa1779a8650", + "accepted_dev": "00d859d9a2e5e104c0950b36c57e097061d6303d", + "revision": "00d859d9a2e5e104c0950b36c57e097061d6303d", + "tree": "1a3499c464341fb103f68a1c1db6a5e5c452a5b1", "prs": [ { - "pr": 75, - "base": "17f48fb932facbdcc45068afd2db6036ac2ec94d", + "title": "Keep header controls on narrow screens", + "number": 44, + "state": "merged" + }, + { + "title": "Prevent region-input focus zoom", + "number": 45, + "state": "merged" + }, + { + "title": "Show advert summaries in packet rows", + "number": 46, + "state": "merged" + }, + { + "title": "Keep the desktop region picker right-anchored", + "number": 47, + "state": "merged" + }, + { + "title": "Observer comparison with dates, overlap counts and percentages", + "number": 48, + "state": "merged" + }, + { + "title": "Node list and detail badges for the optional foreign-repeater flag", + "state": "merged", + "number": 49 + }, + { + "number": 52, + "title": "Traffic trends, hourly heatmap and reception share", + "state": "included via #99" + }, + { + "number": 53, + "title": "Regional scope charts and exact counts", + "state": "merged" + }, + { + "number": 55, + "title": "RF / Signal charts, sample coverage and exact values", + "state": "merged" + }, + { + "number": 57, + "title": "Paths & Hashes charts and exact counts", + "state": "merged" + }, + { + "number": 59, + "title": "Distinct analytics navigation icons", + "state": "merged" + }, + { + "number": 61, + "title": "Exclude reset and invalid node locations from maps and paths", + "state": "merged" + }, + { + "number": 62, + "title": "English and French navigation with matching header dropdowns", + "state": "merged" + }, + { + "number": 63, + "title": "Signal translation", + "state": "included via #99" + }, + { + "number": 64, + "title": "Paths translation", + "state": "included via #99" + }, + { + "number": 65, + "title": "Traffic translation", + "state": "included via #99" + }, + { + "number": 66, + "title": "Scopes translation", + "state": "included via #99" + }, + { + "number": 69, + "title": "Clock Drift and stable table sorting", + "state": "included via #99" + }, + { + "number": 70, + "title": "Shared timestamp translation", + "state": "merged" + }, + { + "number": 68, + "title": "Talkers data state correction", + "state": "merged" + }, + { + "number": 72, + "title": "Mesh data state correction", + "state": "merged" + }, + { + "number": 73, + "title": "Packet summaries in the endpoint cell", + "state": "merged" + }, + { + "number": 91, + "title": "Mirror optional WebSocket repeat and observer-key fields", + "state": "merged" + }, + { + "number": 75, + "title": "fix(packets): show all ambiguous endpoint candidates", + "state": "included via #99", "head": "f4092a492c506c0b81c8067a62c1f7635607b782" }, { - "pr": 79, - "base": "f4092a492c506c0b81c8067a62c1f7635607b782", + "number": 79, + "title": "feat(observers): open a unified monitoring destination", + "state": "included via #99", "head": "ab5bc0c38963cfb91a64b3fb18c8c2e3d64b4bba" }, { - "pr": 80, - "base": "ab5bc0c38963cfb91a64b3fb18c8c2e3d64b4bba", + "number": 80, + "title": "feat(observers): lead with activity and readable health metrics", + "state": "included via #99", "head": "368ba1801681afcf77a65d54a4d411c5962e36ba" }, { - "pr": 81, - "base": "368ba1801681afcf77a65d54a4d411c5962e36ba", + "number": 81, + "title": "feat(observers): compare aligned dashboard activity and retained overlap", + "state": "included via #99", "head": "4b207fafd28c5637177f44f1b0025072ce90cd6d" }, { - "pr": 83, - "base": "4b207fafd28c5637177f44f1b0025072ce90cd6d", + "number": 83, + "title": "feat(packets): connect retained reports to observers and path maps", + "state": "included via #99", "head": "327ea06e92a1fbbdc2674d14ec8386216ce63a2a" }, { - "pr": 85, - "base": "327ea06e92a1fbbdc2674d14ec8386216ce63a2a", + "number": 85, + "title": "feat(routes): inspect retained packets and reporting observers", + "state": "included via #99", "head": "7a82605e8ecbf343bc5f0d9d4b800a43d7243ab5" }, { - "pr": 87, - "base": "7a82605e8ecbf343bc5f0d9d4b800a43d7243ab5", + "number": 87, + "title": "fix(navigation): preserve observer investigation and keyboard return", + "state": "included via #99", "head": "c0c11829bc6013ff90294557c3150844a5cfa09c" }, { - "pr": 89, - "base": "c0c11829bc6013ff90294557c3150844a5cfa09c", + "number": 89, + "title": "feat(channels): show and filter recorded transport scopes", + "state": "included via #99", "head": "19bbbd124982584841e76a4f6e14a61474c87e3a" }, { - "pr": 92, - "base": "19bbbd124982584841e76a4f6e14a61474c87e3a", + "number": 92, + "title": "fix(packets): prevent route and scope overlap", + "state": "included via #99", "head": "fa64057c67fad158072ddf823b0edb0cbc4ad184" }, { - "pr": 95, - "base": "fa64057c67fad158072ddf823b0edb0cbc4ad184", + "number": 95, + "title": "fix(history): align chart periods with retained packet data", + "state": "included via #99", "head": "041a000d9c23c325a65290907fadaa017623132b" }, { - "pr": 99, - "base": "041a000d9c23c325a65290907fadaa017623132b", + "number": 99, + "title": "fix(ui): prepare compact observer monitoring for 1.3.2", + "state": "merged", "head": "eb241ca78b320364aa0eaf1c35e41f45a2362023" + }, + { + "number": 100, + "title": "build(deps-dev): bump brace-expansion from 5.0.9 to 5.0.12", + "state": "merged", + "head": "b4fd148f17ecb220667524efb4fc2bf587df03fb" + }, + { + "number": 101, + "title": "fix: follow up on the #75-#99 stack review", + "state": "merged", + "head": "8c3a58d335ff86306dc684f547ed3364d33e7c34" + }, + { + "number": 102, + "title": "fix: follow up on last week's merges (#55-#101)", + "state": "merged", + "head": "0ac58e1a8bf0e6c0efa953e905db8f54b8b608ed" + }, + { + "number": 103, + "title": "fix(nodes): clear a node's position when nodeUpdate sends null", + "state": "merged", + "head": "c16b2c3ad5e48fb3f5935ba9d444d882f53ef611" } ], - "tests": 955, - "source_sha256": "97f21fb63763ea7e7916cfa7caf24a85eb20e6d1206c01b89af4ace97a036720", - "dist_sha256": "0fdc156ba78a9fdf136a3e128cc3767f2ffda1c20e512a0099d3cdd3a2978ce6", + "tests": 1017, + "source_sha256": "0ff0d89546d520da48dcc992cbb80623856eba1f7d9ab3b12e1596395a0e7e1d", + "dist_sha256": "ea0aa1c67cf4fd8628fcb3fcd7183c9d2767066214a2d2ba2d8a5ed1953a1166", "public_assets": 21 }, "post_release_atlas": { @@ -130,21 +280,19 @@ "parent": "eb241ca78b320364aa0eaf1c35e41f45a2362023", "head": "66ae0cc222a6580b52f62051025453a992956cc6", "tests": 975, - "deployed": false + "deployed": false, + "needs_rebase": true }, "excluded_candidates": { - "server": [ - 182 - ], + "server": [], "web": [ - 93, 97 ] }, "recovery": { - "server_checkpoint": "release132-cutover-20260930T021238Z", - "web_checkpoint": "web-20260930T110037Z", + "server_checkpoint": "latest-cutover-20260930T152333Z", + "web_checkpoint": "web-20260930T155009Z", "config_sha256": "b3d96f52b25e38a0f3ecfb15cd2e492db1d4f494b18774598a4cfad295e2d62f", - "server_binary_sha256": "7f114816f6abb316cf388bfe0f33cd72c1d15cb7ceee50d66f01c8c9b658b76e" + "server_binary_sha256": "2749858070ef9be1cb92757b88f41dfa55fa0d82366eb6b1fc0f906c76058bcc" } } diff --git a/app_documentation/release-132-preparation.md b/app_documentation/release-132-preparation.md index 67edf16..e3b719f 100644 --- a/app_documentation/release-132-preparation.md +++ b/app_documentation/release-132-preparation.md @@ -1,5 +1,24 @@ # Beacon 1.3.2 preparation +## Latest merged candidate — 30 September + +The test site now runs exact merged dev server **689bc232** and web **00d859d9**. Atlas #97 remains excluded and needs a post-release conflict refresh. All release web changes landed through #99, followed by #100–#103; their closed parent PRs are recorded as included via #99 rather than independently merged. Server #182 and the maintainer follow-ups are included. + +Upstream CI, native Go/PostgreSQL tests, the 3,200-input replay, and native web build/lint/**1017 tests** pass. Migration 043 was tested on a restored backup: it cleared 273 stale location records and preserved all raw counts and other located-node fingerprints. Public sources/assets, both MQTT feeds, 26 boundaries, desktop/phone and English/French browser checks pass. Only the Beacon app restarted; the other 23 containers are unchanged. + +Current recovery: `evidence/latest-candidate-20260930/deploy-server.py rollback` on the Pi restores server 397d76b3 / web eb241ca7, preserving newer traffic and the compatible 043 location cleanup. It rolls the frontend back first if necessary. Checkpoint `latest-cutover-20260930T152333Z` is restored/checksummed on and off the Pi. Frontend-only recovery uses the same phase's `deploy-beacon-web.py rollback --evidence-dir latest-candidate-20260930` and checkpoint `web-20260930T155009Z`. + +The initial f6fe177a validation exposed an existing readiness-message race in TestChannelMessageScopeLive; its failed/repeated runs are retained. The current 689bc232 native suite passed. The first replay under concurrent frontend work timed out; the final serialized replay passed in 13.26 seconds with 100 packets, 800 observations, 100 decrypted messages, 800 ordinary / 2,400 opted-in events, and zero fixture drops. These are bounded fixture results, not a claim of universal packet-loss absence or production capacity. + +**Release follow-up:** upstream web #101 restored 24h/7d/30d controls on Observers, Routes and Analytics, and old 3d observer links now select 7d. This differs from the contributor's requested 24h/3d raw-history controls. The test site intentionally matches the exact merged source; this discrepancy remains open for the release decision. Passing tests and smoke checks do not resolve that product requirement. + +Maintainers retain production rollout, physical iPhone/Safari validation and stable release tags. + +## Historical preparation record + +The following records describe the earlier candidate and are superseded by the snapshot above. + + 30 September 2026. Maintainers choose acceptance, the release commit, tags and the production switch. **1.3.2 is the web version**; the server currently has a v1.6.0 release and needs its own version decision. The dev interface reports 1.3.1, while the latest published web GitHub release remains v1.3.0. The release composition stops at [web #99](https://github.com/MeshCore-Beacon/beacon-web/pull/99). [My Atlas #97](https://github.com/MeshCore-Beacon/beacon-web/pull/97) is held for after 1.3.2, as one feature PR based on the release work. Removing Atlas from this preview does not clear saved browser cards. From 528297af88953b8aa3bd719bbb5ac608cac92d60 Mon Sep 17 00:00:00 2001 From: n30nex Date: Wed, 30 Sep 2026 14:40:57 -0400 Subject: [PATCH 40/69] docs(release): prepare Beacon 1.4.0 production cutover --- CONTRIBUTOR_WORKFLOW.md | 12 +- README.md | 8 + RELEASE-CHECKLIST.md | 212 +++++------------- ROADMAP.md | 18 ++ app_config/.env.example | 14 +- .../meshmapper-boundaries-plan.md | 4 +- app_documentation/release-132-preparation.md | 2 + app_documentation/release-140-heads.json | 66 ++++++ app_documentation/release-140-preparation.md | 190 ++++++++++++++++ docker-deployment-type1/docker-compose.yml | 6 +- 10 files changed, 367 insertions(+), 165 deletions(-) create mode 100644 app_documentation/release-140-heads.json create mode 100644 app_documentation/release-140-preparation.md diff --git a/CONTRIBUTOR_WORKFLOW.md b/CONTRIBUTOR_WORKFLOW.md index 9e9692d..28067f8 100644 --- a/CONTRIBUTOR_WORKFLOW.md +++ b/CONTRIBUTOR_WORKFLOW.md @@ -126,9 +126,15 @@ Tests cover squash/drop-parent behavior, a fresh phase after all merges, cache r Markdown is UTF-8. Read and write it with an explicit UTF-8 encoding in scripts, especially when moving between Windows tools. Check both the diff and rendered text before publication. Keep numeric ranges as en dashes and dependency arrows as arrows; do not round-trip the document through a legacy Windows code page. -## 1.3.2 release cut - -Keep the active web manifest through #99 and keep Atlas #97 in a separate post-release manifest. Refresh the entire declared dependency sequence after review edits; publish with the recorded remote heads, then check the actual published revisions. The Atlas manifest can include the verified release parents to reuse identical-tree receipts, but must not change the active preview composition. Build and publish the release web archive without Atlas, preserve the prior archive/assets and browser-local saved cards, and keep Atlas based on the release tip for the owner's next phase. +## 1.4.0 release cut + +Server #189 and web #105 are the active release-preparation PRs on fresh accepted +dev bases. The earlier feature stacks are accepted; do not rebase them again. +Keep Atlas #97 outside the release and in the separate post-1.4.0 manifest. It +needs conflict resolution and fresh validation after the release head is accepted. +Refresh/publish/check the declared active queues when their bases or heads change. +Keep production at live.meshcore.ca pinned to approved releases and dev.meshcore.ca +online for development only. See the [owner cutover plan](app_documentation/release-140-preparation.md). The bounded border snapshot tool is independent of the stack helper and never deploys or changes configuration. The contributor CI runs all `test_*.py` files, including the stack safety checks and exact-member/null/invalid-polygon coverage (26 tests at this checkpoint). diff --git a/README.md b/README.md index 6ab0285..e93ea34 100644 --- a/README.md +++ b/README.md @@ -8,6 +8,12 @@ This repo is the single place to: 2. **Read the docs** — project-wide design and API documentation that describe how the whole system works. 3. **Follow development** — the [parity and analytics roadmap](ROADMAP.md) and [executable contributor workflow](CONTRIBUTOR_WORKFLOW.md) track review dependencies, validation and the next phases. +**Beacon 1.4.0:** [release and cutover plan](app_documentation/release-140-preparation.md). +The intended production destination is `live.meshcore.ca`, replacing CoreScope after +Alderson's approval. `dev.meshcore.ca` stays online for development testing only. +The Canadaverse preview remains the separate review candidate. My Atlas is deferred +until after 1.4.0. These roles do not imply that production has already switched. + --- ## Deploy Beacon @@ -59,6 +65,8 @@ Set every `CHANGE_*` value. The variables you must fill in: | Variable | Service | What to set | |---|---|---| | `POSTGRES_DSN` | `app` | Database connection string. Change the password (`CHANGE_DB_PASS`) to a strong one. | +| `BEACON_SERVER_IMAGE` | `app` | Required reviewed server tag or digest, chosen independently of the web version. | +| `BEACON_WEB_IMAGE` | `web` | Required reviewed web tag or digest; `1.4.0` becomes available after the stable tag is published. | | `REDIS_ADDR` | `app` | `redis:6379` — points the API at the compose Redis service. Leave it out and the server runs uncached, so every read hits Postgres. | | `MQTT_BROKER_1_*` / `MQTT_BROKER_2_*` | `app` | URL, username, and password for your live MeshCore MQTT packet sources. | | `DOMAIN` | `caddy` | Your public domain (e.g. `beacon.example.com`). Caddy auto-provisions a Let's Encrypt cert for it. | diff --git a/RELEASE-CHECKLIST.md b/RELEASE-CHECKLIST.md index 623cdae..d840e3b 100644 --- a/RELEASE-CHECKLIST.md +++ b/RELEASE-CHECKLIST.md @@ -1,155 +1,57 @@ -# Server/web consolidation release - -## 1.3.2 release preparation — 30 September - -The Pi now serves exact merged dev server `689bc232` / web `00d859d9`, with Atlas excluded. The accepted web stack and #100–#103, server #182 and later maintainer fixes are included. Native PostgreSQL tests, the 3,200-input replay, web build/lint/all 1,017 tests and public/browser checks pass. Migration 043 passed on the restored backup without changing raw counts; all 21 assets, source archives and 26 boundaries match. - -My Atlas #97 is held post-release at `66ae0cc2` and needs a conflict refresh against current dev. Web #93 was closed without merge. The known test-readiness race and first replay timeout are preserved in the validation record; the final serialized replay passed in 13.26 seconds. These checks do not certify sustained production capacity. - -[Current audit, exact heads, validation and recovery](app_documentation/release-132-preparation.md). [Live candidate and source](https://canadaverse.org/beacon-dev/source.html). Maintainers retain web acceptance, tags, version decisions and production rollout. Earlier dated records below are historical. - -## Owner release gates for 1.3.2 - -- [x] Accept the server release work and verify the current dev head `689bc232`; CI and native tests pass on the deployed source. -- [x] Accept the release web stack via #99 and follow-ups #100–#103; CI and native tests pass on deployed `00d859d9`. -- [ ] Accept docs #5 so the canonical operator links in server #172/#174 resolve. -- [x] Record the additional dispositions: server #182 is merged and tested, profiling remains disabled on the preview, and web #93 is closed without merge. -- [ ] Resolve the restored 7d/30d UI controls from #101 against the earlier 24h/3d raw-history requirement. Old 3d observer links currently select 7d. -- [ ] Confirm the deployed source/binary pair on the affected MeshMapper host, its packet/summary retention, broker inputs, bounded queues, and private rollback. Apply reviewed border files only where manual boundaries are missing. -- [ ] Compare matched busy periods for accepted/dropped inputs, database work, process CPU/RSS and request latency. The Pi replay and short runtime sample do not certify production capacity. If using #182, keep profiles private and bounded. -- [ ] Check the main operator journeys on desktop and physical iPhone Safari, including English/French, Back, dialogs, maps and expired records. -- [ ] Promote the accepted web source to main and cut **web v1.3.2** under the repository's release process. Choose the server version independently of its existing v1.6.0 tag. Publish matching source and retain rollback artifacts. -- [ ] Leave My Atlas #97 out of the release. Keep its one feature branch based on the accepted release work and refresh it after any squash/rebase merge before post-release acceptance. - -## Historical Atlas preview — 29 September - -[Web #97](https://github.com/MeshCore-Beacon/beacon-web/pull/97), `4fd4b0de`, is the single My Atlas feature PR requested by the contributor. It follows #95 at `e1133ab5` and closes [web #96](https://github.com/MeshCore-Beacon/beacon-web/issues/96) on acceptance. Earlier application PRs remain included; no parent was rebased for this feature. - -My Atlas sits in the desktop tab row and the phone More menu. Visitors save up to twelve full-key node identities, order and a 24h/3d window in this browser. Compact cards show reception bars, SNR/RSSI meters and server freshness; Heard by and statistics expand on demand. Search collapses on return visits. Node, observer/dashboard and exact packet/path investigation reuse the existing navigation. English and French ship together. - -Counts are explicitly the latest **200 retained origin-key reports per node**, filtered to the selected period. Companion requests and other identified-origin packets are included as well as adverts. This is not a complete node-traffic total. Heard by describes the latest loaded packet, not lifetime reach. Missing readings, expired details and incomplete samples remain visible; no radio-health or packet-loss score is invented. - -The Pi now runs unchanged server `a35cba1d` with web `4fd4b0de`. Windows and native Pi build/lint/all **960 tests** pass, along with the actual published-head CI (web CodeQL skipped). Desktop, French 390px phone, keyboard, persistence/order/removal, packet/observer links and dashboard Back checks pass. The public 19 assets and both source archives match, both MQTT feeds are connected, and all 24 container identities/restart counts are unchanged. Physical iPhone Safari and full production capacity remain separate release gates. - -[My Atlas preview](https://canadaverse.org/beacon-dev/?tab=MyAtlas) · [Changelog and source](https://canadaverse.org/beacon-dev/source.html). Frontend rollback restores `e1133ab5` with server `a35cba1d`, from `web-20260930T004937Z`. For an older backend rollback, restore this frontend first, then use the existing September 29 backend recipe; its guard intentionally rejects an unknown newer frontend. - -The contributor explicitly prioritized My Atlas for this phase. Next: refresh maintainer feedback and issues, then server #183 before optional MeshMapper boundaries. Broader issues and remaining parity work stay open. All eighteen application candidates are out of draft; maintainers retain acceptance, merges, stable releases and production cutover. - -See [My Atlas validation and recovery](app_documentation/my-atlas-20260929.md). - -## Earlier integration gate — 29 September - -All six server reviews are addressed in their existing PR sequence. The fixes restore analytics indexes, consolidate unmerged migrations, preserve current partial activity buckets, align cache windows, narrow the route index, keep manual scope priority and simplify channel insertion metadata. Independent server #184 fixes RFC3339 offsets; new web #95 follows #92 and matches time choices to retained data. Existing candidates remain included. - -The Pi runs server `a35cba1d` / web `e1133ab5`. All seventeen published application heads pass Check/CI (web CodeQL skipped); native Go/PostgreSQL and Windows/Pi web build/lint/all 940 tests pass. The restored-copy repair preserved raw rows and archive fingerprints, and rollback index restoration passed. A 3,200-input/504-scope replay had all expected rows/events and zero fixture drops. The one-minute live sample had no parser fallbacks, queue overflows, SQL errors, restarts or reconnects; malformed-IATA and clock-skew warnings remain. - -Visible periods are **24h / 3d** for observer monitoring and route evidence, and **24h / 3d / 30d** for summary-backed Analytics. Seven-day buttons are removed. Raw comparison spans are capped at three days. Durable hourly aggregates already preserve expired packet counts; materialized views combine them with live rows. Older summaries accumulate after archiving starts, and packet detail remains unavailable after expiry. Revised labels and notes are English/French. - -Current acceptance still requires maintainer re-review, especially #167/#169/#174. No upstream merge, stable release or production cutover was performed. The next focused issue is server #183 (saved-route prefix-width changes); broad partial issues remain open. External server #182 and web #93 are unmerged and not in this tested composition. Owners decide the release breakpoint and production switch. - -See [review corrections, source heads and recovery](app_documentation/review-release-20260929.md). No stable release is claimed until maintainers accept the reviewed application heads and choose the release artifacts. - -## Packet reception investigation — 27 September - -[Web PR #83](https://github.com/MeshCore-Beacon/beacon-web/pull/83), `1d5d65e2807c2cb53a998743212d9a5b64ba80c4`, follows #81 and closes focused issue #82 when accepted. It adds grouped retained packet reports, selected-report links, observer inspection/dashboard access and a selected-path map. The initial list stays compact and keeps the selected group open. Equal prefixes are not treated as confirmed identical physical routes; empty/missing paths and TRACE intended routes have explicit labels. - -Map projection omits ambiguous/unlocated identities and breaks lines at gaps. Live animations across uncertain chains are suppressed, so fewer speculative lines appear. Unavailable selected paths no longer silently show All paths. A shared-path loading race is fixed by checking the requested packet hash. Packet labels use the existing Noto Sans stack; external basemap emoji-glyph/sprite fallback warnings can still occur. - -At the packet-investigation checkpoint, the Pi ran web `1d5d65e` with unchanged server `88c2c10c`. Native build/lint and all **906 tests** pass; focused Windows checks and desktop/390px phone/English/French/keyboard/Back/shared-link checks pass. Public assets and source match, both MQTT feeds are connected, and the frontend publication restarted no services. Earlier review candidates remain included. The [changelog/source](https://canadaverse.org/beacon-dev/source.html) identifies the running build. Maintainers still own merges, stable releases and production cutover. - -**Historical follow-up:** route evidence, observer return navigation and MeshMapper scope import were subsequently delivered as review candidates; see the current roadmap. Broader server #60/#72/#99/#116 and web #12 remain open; this is a first connected-investigation slice, not full parity. - -## Observer release checkpoint — 27 September - -The observer-first release is implemented in four focused review candidates: [server #169](https://github.com/MeshCore-Beacon/beacon-server/pull/169) (`91b21995`, closes #168), [web #79](https://github.com/MeshCore-Beacon/beacon-web/pull/79) (`3a0eb4c8`, closes #76), [web #80](https://github.com/MeshCore-Beacon/beacon-web/pull/80) (`b3a6b088`, closes #77) and [web #81](https://github.com/MeshCore-Beacon/beacon-web/pull/81) (`42ae09b7`, closes #78). All are out of draft. Server #169 follows #167; web order is #75 → #79 → #80 → #81. Independent server #166 remains in the preview composition. Maintainers control acceptance and release; these issues remain open until their changes are accepted. - -At the observer-release checkpoint the Pi ran composed server `88c2c10c830034cee70a46fca717af518803a544` and web `42ae09b7c6be50cd0f617bad8aea35825be9f12e`, with the [updated changelog and exact source](https://canadaverse.org/beacon-dev/source.html). Raw packets remain 72 hours, archived hourly analytics 30 days, telemetry 720 hours. All four candidates passed their native Pi suites; the final web build passes 895 tests. Server tests use actual PostgreSQL. Windows server/full destination/dashboard validation and 48 focused final comparison/API tests also pass. Desktop, 390-pixel phone, English/French, keyboard, legacy/shared links and Back/search/sort/scroll were checked. Physical iPhone Safari and production-volume capacity remain separate gates. - -Migration 040 preserves original rows and repairs available archived unknown-payload counts without inventing signal samples. A restored clone passed the migration probe. A fresh private dump was copied off the Pi and its checksum verified; the original schema039 database and matching binary/config/source are retained for DB-aware rollback. Only the Beacon app restarted for the server change; 22 other containers were unchanged. Frontend publication restarted no services. Both MQTT feeds reconnected. Public admin, backup and foreign detection remain disabled. - -Use the [current roadmap](ROADMAP.md) and [observer release contract](app_documentation/observer-monitoring-plan.md) for the current queue. The sections below retain earlier dated validation; their old preview/rollback identities are historical and must not be used as current deployment instructions. - -## September 26 retention and endpoint fixes - -The Pi was first matched to accepted dev server `91b4b457` / web `54b5093a`, including native validation and a verified restore across migrations 037/038. At that checkpoint it ran composed server `a8394f10` and web `3a18e6d1`, adding three focused review candidates: - -| PR | Head | Scope / closure | -|---|---|---| -| [Server #166](https://github.com/MeshCore-Beacon/beacon-server/pull/166) | `74f16de8` | First/renamed adverts resolve after the node update; closes #164 | -| [Server #167](https://github.com/MeshCore-Beacon/beacon-server/pull/167) | `76428b1b` | 30-day hourly summaries survive raw packet expiry; closes #165 | -| [Web #75](https://github.com/MeshCore-Beacon/beacon-web/pull/75) | `3a18e6d1` | Additional-match count and all endpoint candidates on hover, keyboard or touch; closes #74 | - -All are out of draft. Exact-head build CI passes, server CodeQL passes, and web CodeQL remains skipped. MrAlders0n/Claude review was requested in PR comments because formal review requests are unavailable to the contributor account. Both server PRs are independent on the same accepted dev and may merge in either order. Web #75 is also independent. The workflow records #167 plus #166 as a complete PR/head preview input; its separate manifest can be refreshed after upstream changes. No routine manual restacking is required for these non-overlapping changes. No upstream PR was merged by the contributor. - -The agreed Pi policy is **72-hour raw packets, 30-day hourly analytics and 720-hour telemetry**. Migration 039 archives compact summaries as each raw packet cohort expires, atomically with deletion, without storing bodies or raw paths. Traffic, payload, top observers, talkers, advertisers, observer activity, Signal and Paths use the retained summaries. Already-purged history cannot be recovered. Packet detail, sub-hour activity and exact observer comparisons still use retained raw data; entity/scope/radio population counts keep their current meaning. - -Full Windows and native Pi Go/PostgreSQL checks pass, including rollback on archive failure, retries, concurrent ingestion, late observations, distinct observers across batches, multiple IATAs, nullable/radio/path semantics and independent 30-day expiry. The full frontend build/lint and **874 tests** pass. A 1,001-packet fixture with 1KB bodies compacted to at most twelve archive rows; the first Pi run took 85ms for archive/deletion, which is a fixture measurement rather than a production-throughput guarantee. A restored copy of the actual preview database retained all eight view counts after every raw packet was deleted in a rolled-back test. Source/index hashes and the public candidate popup were verified. - -Migration 039 preserved fingerprints of all 23 original application tables. The actual prior schema038 database, exact binary/configuration and private dump remain available for rollback; older pre-038 recovery is retained separately. Only the Beacon preview app restarted (about 33 seconds); 22 other containers were unchanged and both MQTT feeds reconnected. Public admin/backup and foreign detection remain disabled. [Current changelog and corresponding source](https://canadaverse.org/beacon-dev/source.html). - -Current broader issues remain server #60 (admin), #72 (backup/import), #99 (packet summaries), #116 (MQTT investigation), and web #12 (remaining translations). Prioritize feedback and acceptance of these new fixes, then a focused Mesh/Talkers/Observer translation slice under #12. A measured month of accumulated history, production-scale capacity and physical Safari checks remain separate gates. Maintainers own stable releases and the owners handle production cutover. - -## Historical consolidation scope (24 September) - -Release the accepted account/backup/analytics batch before Channel Activity or further parity expansion. Current public tags are server v1.6.0 and web v1.3.0; maintainers choose the next versions and perform signed release commits, main promotion and tags under each repository's contribution rules. - -The deployment owner performs the eventual CoreScope switch. Beacon remains at dev.meshcore.ca, CoreScope at live.meshcore.ca, and the Pi preview remains at canadaverse.org/beacon-dev/ with its changelog and corresponding source. - -The release candidate includes both formerly independent follow-ups: [server #160](https://github.com/MeshCore-Beacon/beacon-server/pull/160) for offline archive verification and [server #161](https://github.com/MeshCore-Beacon/beacon-server/pull/161) for ACK/TRACE/PING summaries. Their wider issues #72 and #99 remain partial. Integrity checks do not establish archive authenticity or restorability; packet references do not establish identity or delivery. - -Accepted server: `c02317a4ac7228d19cab498edfa1d61186c84626`. -Accepted web: `0f0a6ca51c7b2c3315db77954f61b30bbdeea5e2`. -The web source tree is identical to tested preview `42ba5fcb`; preserve that artifact's actual revision/source instead of relabeling it. The server differs from preview `5848d200` only in the verifier CLI/library/tests/docs; its verifier code and tests are identical to separately tested `262eae96`. The documentation additionally contains the accepted protected-download section. - -## Historical review gates (24 September) - -- [x] Workflow checks include every independent preview PR, not just the ordered stack. Status reports them; Check verifies their CI and source; Refresh/Publish reject changed prepared inputs. Server #161 and web #61 are covered by the normal preview checks. The standalone backup CLI #160 is checked with its separate manifest. -- [x] Server #149: document POST/DELETE browser preflights and the full admin CORS method example. Keep public read-only defaults. -- [x] Server #154: verify pg_dump/server compatibility at startup; an optional backup prerequisite failure disables only backup, with a specific operator diagnostic. Document backup.enabled and distinguish the export size limit. Native testing caught and fixed the text-versus-integer version-setting scan; CI now covers it with PostgreSQL. -- [x] Server #157: serve Signal distributions and weighted means from compact materialized data; snap polling windows to hours, preserve missing/invalid/legacy sample semantics, and measure refresh/storage costs. -- [x] Server #159: materialize path classification, share window parsing, guard database-derived array indexes and retain all 256 decoder-header checks. -- [x] Web #60: shared map-location validation in independent #61 omits reset/invalid markers and links while preserving valid zero-axis locations and stored records. Native and real-data browser checks pass. -- [x] Web #58: distinct navigation glyphs in #59, plus dedicated RF/Signal and Paths glyphs in #55/#57. -- [x] Refresh #55/#57 after the accepted #52/#53 squash. Their source trees were identical after the September 20 refresh; that history-only update needs no replacement Pi artifact. -- [x] All eight application PRs pass their required checks on the published heads. Native PostgreSQL tests ran. The upstream web CodeQL job remains skipped under its existing policy and is not counted as a scan. - -All original six server and four web PRs are merged. The September 25 translation batch is accepted through web #70, with #63/#64/#65/#66/#69 closed as included; #68/#72 also merged and #67/#71 are closed. New server #162/#163 and web #73 are accepted. At that checkpoint only docs #5 remained open. Active application manifests are empty, acceptance history remains, and no application branch was rewritten. - -## Storage and retention boundary - -The September 17 drop-and-reset observation-partitioning design and implementation plan were explicitly superseded on September 19. They are historical reference only. Do not implement their table drop, history reset or process-local dedup replacement. - -Accepted server #162 now supplies batched retention deletes, per-table autovacuum tuning in migration 037 and a seven-day default packet/chat retention when unset. #163 adds migration 038, dropping per-observation endpoint snapshots and resolving against current nodes at read time. The earlier partition/reset proposal remains superseded. Compression changes were not added by these two migrations. - -A consolidation release must document its actual retention behavior and capacity limits. A future production parity cutover also needs verified durable history coverage and recovery copies. Do not infer either from example configuration or the Pi's short history. - -Current dev is server `91b4b457` / web `54b5093a`, with passing CI/image builds (server coverage/CodeQL pass; web CodeQL skipped). The Pi still runs server `c02317a4` / web `9d96b943`, equivalent to web source accepted through #72, with prior 867-test native evidence. It lacks server #162/#163 and web #73. The current frontend rollback is `300ee974`. No new deployment/migration occurred in the September 26 audit. Before upgrading, verify a database recovery checkpoint and explicit retention policy: old server queries reference the column removed by 038, so restoring only the old binary afterward is not a valid rollback. - -## Accepted-dev verification - 24 September - -- [x] Exact dev CI/image builds pass at server `c02317a4` and web `0f0a6ca5`; server CodeQL/coverage pass and web CodeQL remains skipped. -- [x] Server `c02317a4` built/tested natively on the Pi with real PostgreSQL: 1,194 passing test/subtest results. Signal, Paths, observer comparison, migration recovery, packet summaries and NULL observations ran. Two opt-in backup export/download integration suites were skipped; prior private restore/TLS checks remain separately dated evidence. -- [x] Accepted server is running on the preview. Source/asset hashes match; both feeds advance. Signal/Paths reconcile with SQL for global/regional 1/7/30-day selections, at 2-19 ms origin latency. Only three complete hours are populated; this does not prove 7/30-day history coverage. -- [x] Browser Signal/Paths charts and map load with LIVE status and no captured warnings/errors. Unchanged frontend assets keep their actual `42ba5fcb` build/source identity and prior 786-test evidence; accepted `0f0a6ca5` has the identical tree. -- [x] Only the Beacon app restarted; the other 22 containers and configuration/migration journal were preserved. Immediate rollback is server `5848d200` with unchanged web. The separate verifier retains its real `262eae96` binary/source identity. -- [x] Accepted review queues/overlays were retired, including the translation ancestors verified as included in #70. All application PRs are accepted. Only docs #5 and the five broader issues remain open. - -## Earlier combined-candidate evidence - 20 September - -- [x] September 20 unmodified Pi stability sample: 600 seconds / 41 samples, both feeds connected, 2,169 retained observations and no MQTT disconnect/deadline or HTTP 5xx. App/PostgreSQL CPU averaged 2.14%/3.96% of one core. This is a bounded health sample, not callback timing, #116 root-cause proof or a production-volume gate. [Result and limits](https://github.com/MeshCore-Beacon/beacon-server/issues/116#issuecomment-5753626821). -- [x] Native Pi build/test of server `6be0f762` and web `42ba5fc`, including real PostgreSQL-to-HTTP checks and migration retry/concurrent refresh; 786 web tests pass. -- [x] One-million-row request/initial-population/refresh/storage measurements; request plans read only the new views. Signal: 1.8–77.5 ms reads, 14.4 s initial population, 16.8 s refresh, 6.5 MB. Paths: 3.5–141.9 ms reads, 8.4 s population, 6.2 s refresh, 11.3 MB. -- [ ] Measure the full operator workload and sustained refresh/ingest load before a production parity claim. The million-row fixture does not establish that limit. -- [x] Backup client mismatch and unsupported-DSN cases leave the public API available; valid client export/restore used disposable data only and restored all 35 source migrations. Public preview admin/backup remains disabled. -- [x] Real preview analytics reconcile with SQL for the materialized window. Browser charts, complete-hour text, small screens and error/empty/retry states are verified. Both MQTT feeds advance and the public browser reports LIVE. -- [x] Current and rollback server/web artifacts, exact source offers and visible changelog match the running pair. Only the Beacon app restarted; the other 20 containers were preserved. Additive rollup migrations retain observations and are compatible with the previous binary. - -The historical pre-retention rollback restored combined frontend `300ee974` with server `c02317a4`. Restore accepted frontend `42ba5fcb` before using the older consolidation server rollback to `5848d200`; its metadata describes the accepted frontend. The September 20 packet-reference rollback to `6be0f762` is an older recovery point. Exact artifacts/runners are retained; application rollback keeps additive rollup views and does not remove history. - -## Maintainer release handoff - -1. Review current server #166/#167/#169 and web #75/#79/#80/#81. Preserve dependencies; choose an explicit release freeze and validate its actual heads. Ready for review is not owner approval or a published release. -2. Review migration 039/040 and the verified database-aware recovery boundary. Deploy server metrics before dependent observer pages; retain packet/summary counting definitions and approved retention settings. -3. Follow each contribution guide for signed version/API commits, main promotion, tags and release CI. Reconcile stable web history instead of overwriting main. No versions or tags were chosen by this contribution. -4. Verify exact Actions-built release artifacts, corresponding source, upgrade/retention guidance and rollback on the intended deployment. Owners perform the eventual production switch. -5. Keep broader #60/#72/#99/#116 and web #12 open for their remaining scope. Continue connected investigations after feedback, with the scope-import draft separate. Physical Safari, sustained production workload and a measured month of retained history remain validation gates. +# Beacon 1.4.0 release checklist + +1.4.0 replaces the planned 1.3.2 release. Alderson controls acceptance, stable tags +and the production switch. Earlier receipts remain in the [historical preparation +record](app_documentation/release-132-preparation.md) and dated evidence documents. + +## Destinations and scope + +- `live.meshcore.ca`: production Beacon 1.4.0, replacing CoreScope after approval. +- `dev.meshcore.ca`: kept online for development testing only, with separate + application, database, cache and configuration. +- `canadaverse.org/beacon-dev/`: the exact contributor review candidate with source + downloads and rollback. It is not the production deployment. +- My Atlas #97: held until after 1.4.0; resolve conflicts and validate it separately. +- Server releases keep their independent version history. Do not downgrade the + existing v1.6.0 server version to web 1.4.0. + +## Candidate verification + +- [x] Web package and lockfile identify 1.4.0; dependencies are unchanged by the bump. +- [x] Development, main and prerelease events cannot publish `latest`; eight official + metadata-action v5 fixture cases pass across both application workflows. +- [x] Both deployment templates accept explicit production/development image references + and matching URLs; missing image selection fails before deployment. +- [x] Native server/PostgreSQL tests, restored-copy migration and bounded ingestion + replay pass. These checks do not establish production capacity. +- [x] Source/assets, public reads, fresh packets and 26 boundaries are verified on + the review preview. Active revisions and the frontend receipt are in the + [candidate manifest](app_documentation/release-140-heads.json). +- [x] A restored/checksummed private dump and prior application/configuration/assets + are retained. Unrelated Pi services are preserved. + +## Alderson's release gates + +- [ ] Review/accept server #189, web #105 and docs #5. Verify CI on the actual + accepted commits, including changes after the recorded candidate. +- [ ] Resolve the earlier 24h/3d raw-history requirement against upstream's restored + 7d/30d controls and the conversion of old 3d observer links to 7d. +- [ ] Verify desktop and physical iPhone Safari, English/French, keyboard, Back, + maps and expired-history states on the production candidate. +- [ ] Stage independent production Beacon data/configuration; restore-test its backup + and migrations and record available history. Do not run Beacon migrations against + CoreScope's database or share the development database. +- [ ] Validate the intended production host over a representative busy period for + ingestion, database load, CPU/memory and latency. Keep profiles private. +- [ ] Promote accepted release source under the repository process, publish web + `v1.4.0`, choose the server version independently, and verify Actions-produced + images, revision labels, immutable digests and corresponding source. +- [ ] Save/verify the CoreScope rollback route, images, configuration and data. + After approval, switch `live.meshcore.ca` web/API/WebSocket traffic together to + Beacon and preserve the CoreScope rollback window. +- [ ] Confirm production version/digests, fresh packets, brokers, TLS, API/WebSocket + destinations and boundaries. Independently verify `dev.meshcore.ca` remains + online and uses only its development backend/data. + +[Detailed cutover and rollback](app_documentation/release-140-preparation.md) · +[Review source/changelog](https://canadaverse.org/beacon-dev/source.html). diff --git a/ROADMAP.md b/ROADMAP.md index da956b0..9c464af 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,5 +1,23 @@ # Beacon parity and analytics roadmap +## Beacon 1.4.0 candidate — 30 September + +**1.4.0 replaces the planned 1.3.2 release.** Alderson reviews and releases it, then switches `live.meshcore.ca` from CoreScope to production Beacon. `dev.meshcore.ca` stays online for development testing only, with separate application/data/configuration and matching development endpoints. Neither host was changed by this preparation. The server retains its independent version history. + +The review preview is server **9054acd8 / web 23945d59**, from server #189 and web #105. The frontend displays **1.4.0**. Atlas #97 remains excluded until after 1.4.0 and requires conflict resolution against the accepted release head. The active queues contain only these two release-preparation PRs; accepted parents are not restacked. The separate future Atlas manifest is `planning/review-stack-web-post140.json`. + +Published-head CI, native Go/PostgreSQL tests, restored-copy migration 044, the 3,200-input replay, and native web build/lint/**1029 tests** pass. Eight actual metadata-action tag cases and six Compose rendering cases pass. Development pushes cannot publish `latest`; stable release tags own it, and production deployment inputs must be pinned. All 20 public assets, both sources, 26 boundaries and desktop/phone English/French checks pass. The server update preserved the other 23 containers; the frontend preserved all 24. + +Backend recovery is `evidence/release-140-final-20260930/rollback.py`, checkpoint `release140-cutover-20260930T174802Z`, restoring 689bc232 / 00d859d9 while preserving new traffic and the compatible additive boundary table. It restores the frontend first if needed. Frontend-only recovery is `evidence/release-140-final-20260930/deploy-beacon-web.py rollback --evidence-dir release-140-final-20260930`, checkpoint `web-20260930T183247Z`. A restored/checksummed private dump is retained on and off the Pi. Configuration remains b3d96f52 / mode 0644; automatic zones, public admin/backup and foreign classification stay disabled. + +**Remaining owner gates:** review/acceptance, the 24h/3d raw-history requirement versus upstream 7d/30d controls, physical Safari, production-host capacity/data verification, approved Actions images/tags and the CoreScope route switch. The bounded replay and brief live sample do not certify production capacity. The earlier readiness-test race remains historical evidence. + +[Current release/cutover plan](app_documentation/release-140-preparation.md) · [Exact candidate record](app_documentation/release-140-heads.json). + +The final web cutoff includes #106/#107; later development is outside this recorded candidate until reviewed. + +## Historical 1.3.2 preparation + ## 1.3.2 release preparation — 30 September The Pi now serves exact merged dev server `689bc232` / web `00d859d9`, with Atlas excluded. The web stack landed through #99, followed by #100–#103. Server #182 and the maintainer ingestion, caching and location fixes are included. Native PostgreSQL tests, the 3,200-input replay, and web build/lint/all **1,017 tests** pass. All 21 public assets, both source archives and 26 boundaries match. diff --git a/app_config/.env.example b/app_config/.env.example index 54c0004..a135c47 100644 --- a/app_config/.env.example +++ b/app_config/.env.example @@ -7,6 +7,11 @@ # ============================================================ # ---- Backend API server (service: app) ---------------------- +# Choose reviewed images explicitly. Use immutable digests for production. +# The server has an independent release version; do not use web's 1.4.0 tag for it. +BEACON_SERVER_IMAGE=ghcr.io/meshcore-beacon/beacon-server:CHANGE_REVIEWED_SERVER_TAG +BEACON_WEB_IMAGE=ghcr.io/meshcore-beacon/beacon-web:1.4.0 + # Address the Go server listens on inside its container. LISTEN_ADDR=:8080 @@ -30,7 +35,7 @@ MQTT_BROKER_2_PASSWORD=CHANGE_PASS # ---- Public domain / TLS (service: caddy) ------------------- # Site address + automatic Let's Encrypt certificate. -DOMAIN=dev.meshcore.ca +DOMAIN=live.meshcore.ca # ---- Frontend URLs (service: web) --------------------------- # Baked into the static JS bundle when the web container starts. @@ -39,8 +44,11 @@ DOMAIN=dev.meshcore.ca # After changing these, recreate the web container so they re-inject: # docker compose up -d --force-recreate web # (and hard-refresh / incognito, since /assets/* is cached immutable) -VITE_API_BASE=https://dev.meshcore.ca/api/v1 -VITE_WS_URL=wss://dev.meshcore.ca/ws +VITE_API_BASE=https://live.meshcore.ca/api/v1 +VITE_WS_URL=wss://live.meshcore.ca/ws + +# For development, use a separate directory/database/cache with DOMAIN=dev.meshcore.ca, +# matching dev.meshcore.ca REST/WebSocket URLs and both image references ending in :dev. # Default "All" map view (decimal "lat,lon" + zoom). Unset = world view. VITE_MAP_CENTER=52.5,-96.8 diff --git a/app_documentation/meshmapper-boundaries-plan.md b/app_documentation/meshmapper-boundaries-plan.md index d05e45c..5708b2b 100644 --- a/app_documentation/meshmapper-boundaries-plan.md +++ b/app_documentation/meshmapper-boundaries-plan.md @@ -1,6 +1,8 @@ # Optional MeshMapper boundary synchronization -A reviewed snapshot workflow is available for the 1.3.2 preparation; see [installation and recovery](meshmapper-border-snapshots.md). Automatic synchronization remains a later phase from the maintainer discussion supplied on 27 September. The [published Zones API](https://wiki.meshmapper.net/zones-api/) already supplies the required catalogue and polygons; no scraping or new endpoint is needed. +Automatic synchronization landed in server dev `0e242574` and is included in the [1.4.0 candidate](release-140-preparation.md). It is independently opt-in through `meshmapper.zones.enabled`, refreshes at a minimum of one hour and uses migration 044's separate boundary table. Imported boundaries override manual shapes; manual `borderFile` remains the fallback where the API supplies no boundary. The contributor preview still uses its reviewed snapshot without enabling this importer. See [snapshot installation and recovery](meshmapper-border-snapshots.md). + +The following paragraphs preserve the earlier design discussion; the accepted override order above supersedes the proposed manual-first order below. The [published Zones API](https://wiki.meshmapper.net/zones-api/) supplies the catalogue and polygons; no scraping or new endpoint is needed. Verified public reads: `https://meshmapper.net/get_zones.php?country=CA` lists enabled regions, and `https://yow.meshmapper.net/get_geojson.php` returns YOW's FeatureCollection. Both are unauthenticated and support ETag caching with a one-hour minimum polling period. Coordinates use longitude, latitude. A missing boundary is explicitly null; group collections contain separate member features. Region codes can be 2–6 alphanumeric characters, so check compatibility with Beacon identifiers before importing or mapping them. Never truncate codes. diff --git a/app_documentation/release-132-preparation.md b/app_documentation/release-132-preparation.md index e3b719f..9e8f95f 100644 --- a/app_documentation/release-132-preparation.md +++ b/app_documentation/release-132-preparation.md @@ -1,5 +1,7 @@ # Beacon 1.3.2 preparation +**Historical proposal:** the release is now [Beacon 1.4.0](release-140-preparation.md). The source and recovery records below describe earlier checkpoints. + ## Latest merged candidate — 30 September The test site now runs exact merged dev server **689bc232** and web **00d859d9**. Atlas #97 remains excluded and needs a post-release conflict refresh. All release web changes landed through #99, followed by #100–#103; their closed parent PRs are recorded as included via #99 rather than independently merged. Server #182 and the maintainer follow-ups are included. diff --git a/app_documentation/release-140-heads.json b/app_documentation/release-140-heads.json new file mode 100644 index 0000000..831421e --- /dev/null +++ b/app_documentation/release-140-heads.json @@ -0,0 +1,66 @@ +{ + "release": "Beacon/web 1.4.0", + "verified_at": "2026-09-30T18:37:14.975711+00:00", + "release_published": false, + "production_switched": false, + "roles": { + "production": "https://live.meshcore.ca/", + "development": "https://dev.meshcore.ca/", + "review": "https://canadaverse.org/beacon-dev/" + }, + "server": { + "pr": 189, + "revision": "9054acd89f96fa3e117db954c384b778a33f7aad", + "base": "0e242574fb5ff852412c8c741877e104c59c4c6d", + "tree": "81335e7cdb2d9e19d81d547ffa2001e959f73782", + "binary_sha256": "870773fa64a8ce7179a1afda21e69ed830f331eaa56a75679bbadc45c1fb4fd9", + "source_sha256": "954bd52cebf9cd5024eb21400004e1e128c6a8e855ceb2d6f2f334eb32e6fba1", + "version_policy": "Independent server version; do not downgrade to web 1.4.0." + }, + "web": { + "pr": 105, + "revision": "23945d5907b031345f8bc5e85b926532272d238b", + "base": "b1f41dea1d471a2d26941401cbe1b2d446995754", + "tree": "59293351133115673dd1770ba03cb4c5067482b6", + "version": "1.4.0", + "tests": 1029, + "source_sha256": "83b72731fc6c0dadf1b9cbd270a43fc9ccb2a66be869efb8caf7fce320344e38", + "dist_sha256": "b92255556988a538cd8961d44b3dfe53c9527de8e3d5bb4722cf40bfe547174b", + "index_sha256": "134093571a33bf3209ade58face8f38114a08f573fe8a4c6629d498a2df5172a", + "public_assets": 20 + }, + "atlas": { + "pr": 97, + "head": "66ae0cc222a6580b52f62051025453a992956cc6", + "hold": "After 1.4.0", + "deployed": false, + "needs_rebase": true + }, + "validation": { + "native_postgres": true, + "restored_migration": "044_meshmapper_zones.sql", + "replay_inputs": 3200, + "replay_fixture_drops": 0, + "image_tag_cases": 8, + "compose_cases": 6, + "public_boundaries": 26, + "windows_and_pi_web_tests": 1029 + }, + "recovery": { + "server_checkpoint": "release140-cutover-20260930T174802Z", + "web_checkpoint": "web-20260930T183247Z", + "config_sha256": "b3d96f52b25e38a0f3ecfb15cd2e492db1d4f494b18774598a4cfad295e2d62f", + "server_recipe": "evidence/release-140-final-20260930/rollback.py", + "web_recipe": "evidence/release-140-final-20260930/deploy-beacon-web.py rollback --evidence-dir release-140-final-20260930", + "frontend_only_restores": "79c09864bdf671ab19d203c07c60f6d23091c3d2 with server 9054acd8" + }, + "remaining": [ + "Alderson review/acceptance and exact accepted-head CI", + "24h/3d raw-history requirement versus upstream 7d/30d controls", + "Physical iPhone Safari and production-host load/data validation", + "Stable image/tag publication and owner-controlled CoreScope cutover" + ], + "web_cutoff": "Accepted web #107 at b1f41dea; later changes require a release decision.", + "web_validation_evidence": "release-140-final-20260930", + "server_validation_evidence": "release-140-20260930" +} diff --git a/app_documentation/release-140-preparation.md b/app_documentation/release-140-preparation.md new file mode 100644 index 0000000..6565e11 --- /dev/null +++ b/app_documentation/release-140-preparation.md @@ -0,0 +1,190 @@ +# Beacon 1.4.0 release and CoreScope cutover + +The planned Beacon/web release is **1.4.0**, replacing the earlier 1.3.2 proposal. +Alderson reviews and releases the candidate, then controls the production switch. +This document prepares that switch; it does not claim that either MeshCore Canada +host has already changed. + +## Destinations + +| Destination | Intended role | Update policy | +|---|---|---| +| `https://live.meshcore.ca` | Production Beacon 1.4.0, replacing CoreScope after approval | Reviewed release images pinned by digest; changes require an owner rollout | +| `https://dev.meshcore.ca` | Development testing only, kept online | Separate application, database, cache and saved configuration; deliberate updates from `dev` | +| `https://canadaverse.org/beacon-dev/` | Contributor review candidate | Exact PR revisions, corresponding source and retained rollback; not the production endpoint | + +Production REST and WebSocket traffic must terminate at the production Beacon +backend. Do not point the production frontend at `dev.meshcore.ca`. A development +restart, migration, test or data reset must not alter production data or routing. +Use separate deployment directories and persistent storage. If environments share +a host, review the existing proxy, internal port bindings and available capacity +before starting a second stack; the Type 1 defaults assume one standalone stack. + +## Review candidates + +| Repository | Candidate | Scope | +|---|---|---| +| Server | [#189](https://github.com/MeshCore-Beacon/beacon-server/pull/189), `9054acd89f96fa3e117db954c384b778a33f7aad`, based on `0e242574` | Stable image publishing and rollout documentation; includes the accepted Zones API importer and migration 044 | +| Web | [#105](https://github.com/MeshCore-Beacon/beacon-web/pull/105), `23945d5907b031345f8bc5e85b926532272d238b`, based on `b1f41dea` | Package/lock version 1.4.0, explicit deployment image selection and stable publishing; includes accepted Analytics #104 and phone/header/channel polish #106/#107 | +| Docs | [#5](https://github.com/MeshCore-Beacon/beacon-docs/pull/5) | Environment roles, pinned deployment inputs, release checklist, validation and recovery | +| Web, deferred | [My Atlas #97](https://github.com/MeshCore-Beacon/beacon-web/pull/97) | Excluded until after 1.4.0; retains its single feature PR and saved-card design, but requires conflict resolution against current dev | + +The server has an independent release history with **v1.6.0 already published**. +1.4.0 identifies the Beacon/web release; do not retag or downgrade the server. +Record its approved revision, independently chosen stable version and image digest +alongside the web release. The mobile repository is outside this release's scope. + +Accept the reviewed server/web changes, verify the resulting CI heads, and accept +docs #5 so the canonical operator-guide links resolve. New upstream changes after +the validation snapshot require a diff review and applicable checks before they +join the production candidate. No Atlas changes belong in either release artifact. + +## Release notes draft + +- Observer monitoring combines packet activity, device telemetry and comparison, + with the observer list alongside the dashboard and an Observer view in Analytics. +- Packet, observer, node, route and map links provide retained evidence for + investigation. Endpoint and path ambiguity remain visible rather than implying + a uniquely identified sender, relay or delivery path. +- MeshMapper scope metadata and optional boundary imports improve regional views; + configured fallbacks and missing/unknown scope states remain explicit. +- Hourly analytics preserve summaries after raw packet expiry. Raw packet detail + remains limited by configured retention; expired observations are not reconstructed. +- English/French navigation, compact phone controls and on-demand packet maps + improve presentation. Atlas remains outside this release. +- Ingestion, route maintenance, cache invalidation, location reset handling and + deployment image separation include the accepted reliability/performance fixes. + Production capacity still needs validation on its intended host. + +## Image publishing and configuration + +Both application repositories currently have `dev` as their default branch. The +release PRs remove the rule that published `latest` from that default branch. +The official metadata-action v5 bundle was exercised with these events for both +repositories; all eight cases passed without publishing any images: + +| Event | Published tags | +|---|---| +| Push to `dev` | `dev`, revision tag | +| Push to `main` | revision tag | +| Stable semantic-version tag | full version, major/minor, `latest`, revision tag | +| Prerelease tag | prerelease version, revision tag; no `latest` | + +Existing registry tags are not rewritten by merging the workflow. The owner must +verify the new stable Actions run, its source revision and image digest before +using its output. A local Pi binary or frontend build is preview evidence, not a +substitute for the approved production release artifacts. + +The complete Type 1 deployment now requires `BEACON_SERVER_IMAGE` and +`BEACON_WEB_IMAGE`. Select the exact reviewed tags for staging, then pin the +published digests for production. Do not use `latest` or `dev` as a production +deployment input. The web repository's standalone `docker/` folder serves only +the frontend; use the complete Type 1 proxy or explicitly configure production +`/api/*` and `/ws` routing to the matching backend. + +| Setting | Production | Development | +|---|---|---| +| `DOMAIN` | `live.meshcore.ca` | `dev.meshcore.ca` | +| `VITE_API_BASE` | `https://live.meshcore.ca/api/v1` | `https://dev.meshcore.ca/api/v1` | +| `VITE_WS_URL` | `wss://live.meshcore.ca/ws` | `wss://dev.meshcore.ca/ws` | +| `BEACON_WEB_IMAGE` | Approved 1.4.0 release image digest | Reviewed `dev` build or its digest | +| `BEACON_SERVER_IMAGE` | Approved matching server image digest | Reviewed `dev` build or its digest | +| Database/cache/configuration | Production-owned copies | Development-owned copies | + +Frontend URLs are injected into the published web image at startup. Recreate +the frontend when changing them and verify the browser's actual destinations. +Review saved CORS and WebSocket origin settings for the appropriate host and the +existing trusted-proxy chain. Keep credentials and private connection strings in +the operator's secret/configuration store, not this document or the release notes. + +## Data continuity and migration + +Use a separate production Beacon database. The proposed rollout seeds it from a +fresh, restore-tested snapshot of the approved Beacon dataset, then validates and +starts its own ingest. The original development database remains with development. +An empty production start is an explicit alternative for Alderson to approve, +with its cold-start and history limits communicated before cutover. + +CoreScope's schema is not a Beacon migration source. Preserve a separate CoreScope +backup and its configuration/images for recovery; no automatic CoreScope history, +settings or saved-node conversion is included. Record the earliest available raw +packets and summaries in the production copy instead of promising unavailable +history. The historical counter discrepancy is not resolved by changing products. + +The agreed candidate policy remains **72-hour raw packets, 30-day hourly summaries +and 720-hour telemetry**. Migration 043 clears stale zero/omitted advert positions; +044 adds separate imported zone-boundary storage. Test migrations on a restored +copy before applying them to the production Beacon database. Verify raw counts, +retained summaries and unaffected node locations. Keep the pre-change dump, +application images, configuration and proxy route available for rollback. + +The new Zones API importer is separately opt-in. When enabled, imported boundaries +override manual `borderFile` shapes; manual shapes remain the fallback where the +API has none. The contributor preview keeps its 26 reviewed manual boundary files, +scope importer and Public channel configuration. Automatic zones, public admin, +public backup and foreign-node classification are not enabled by this release prep. + +## Owner cutover sequence + +1. Freeze the accepted server/web revisions and verify their CI. Publish web + `v1.4.0` through the repository's release workflow, choose the server version + independently, and record both image digests and matching source offers. +2. Save the current `live.meshcore.ca` CoreScope routing, configuration, images and + database backup. Confirm the rollback route before changing public traffic. +3. Stage the production Beacon pair behind the existing ingress without replacing + CoreScope yet. Restore the approved Beacon snapshot into its separate database, + apply and validate migrations, and verify both broker inputs and retention. +4. Check the production candidate's observer dashboard/comparison, Analytics, + packets, channels, nodes, routes, traces and maps. Check English/French, desktop, + physical iPhone Safari, keyboard access and Back behavior. Verify the candidate's + requests reach its own REST/WebSocket endpoints and that older CoreScope browser + sessions load the new assets correctly. +5. Compare a representative busy period for accepted/dropped input, database load, + process CPU/memory and request latency. Fixture replay and a short healthy Pi + sample do not establish production capacity. Keep any profiling private and bounded. +6. After Alderson's approval, switch the `live.meshcore.ca` web/API/WebSocket route + together to Beacon. Keep CoreScope available for the agreed rollback window. + Leave `dev.meshcore.ca` serving its development stack; do not redirect it to live + or remove its service while cutting over production. +7. Verify public TLS, version 1.4.0, exact deployed digests/source, API reads, live + subscriptions, fresh packet arrival and working regional outlines. Verify dev + independently remains reachable and uses only its development backend/data. +8. If acceptance fails, restore the saved CoreScope route and its matching backend. + Preserve the new Beacon database for diagnosis; do not restore an older dump over + newly received traffic. Binary-only Beacon rollback needs a reviewed compatible + schema; otherwise use the retained separate database checkpoint. + +## Validation and remaining release decisions + +The exact PR heads are deployed at the review site and its footer shows 1.4.0. +The web cutoff is accepted #107 at `b1f41dea`; the refreshed web #105 is +`23945d5907b031345f8bc5e85b926532272d238b`. Windows and Pi tests match. +Published-head CI, the native PostgreSQL suite (including zone-boundary storage), +restored migration 044 and the 3,200-input replay pass. The replay retained 100 +packets / 800 observations / 100 decrypted messages and the expected 800 ordinary +and 2,400 opted-in events, with zero fixture drops. Native frontend build/lint and +all **1029 tests** pass. Eight tag-generation cases, six configuration-rendering +cases, public asset/source/boundary checks and desktop/phone English/French browser +checks pass. See the [exact candidate record](release-140-heads.json). + +The current private checkpoint is `release140-cutover-20260930T174802Z`. +The combined Pi recovery is `python3 evidence/release-140-final-20260930/rollback.py`. +It first restores frontend 79c09864, then invokes the guarded backend recovery to +restore server 689bc232 / web 00d859d9. New traffic and the compatible additive +044 table are preserved. Do not skip the newest frontend rollback when using +an older phase's recovery script. + +Frontend-only recovery is `python3 evidence/release-140-final-20260930/deploy-beacon-web.py rollback --evidence-dir release-140-final-20260930`, checkpoint `web-20260930T183247Z`; +it keeps server 9054acd8 and restores frontend 79c09864. The private dump was +restored and checksum-verified on and off the Pi. The server update left 23 other +containers unchanged; each frontend publication left all 24 unchanged. + +The current upstream UI has 24h/7d/30d controls, including raw-evidence views, and +old 3d observer links select 7d. This conflicts with the earlier requested 24h/3d +raw-history controls. Alderson must resolve that requirement or explicitly accept +the changed behavior before release; the version change does not resolve it. + +The existing test readiness-message race, any native validation retries and replay +timing limits remain recorded. Production-host capacity, the actual CoreScope +cutover and physical Safari validation stay explicit owner acceptance steps. +The earlier [1.3.2 preparation record](release-132-preparation.md) is historical. diff --git a/docker-deployment-type1/docker-compose.yml b/docker-deployment-type1/docker-compose.yml index 5e728e8..4abdeff 100644 --- a/docker-deployment-type1/docker-compose.yml +++ b/docker-deployment-type1/docker-compose.yml @@ -1,6 +1,6 @@ services: app: - image: ghcr.io/meshcore-beacon/beacon-server:latest + image: ${BEACON_SERVER_IMAGE:?Set BEACON_SERVER_IMAGE to the reviewed server tag or digest} env_file: .env ports: - "127.0.0.1:8080:8080" @@ -63,7 +63,7 @@ services: restart: unless-stopped web: - image: ghcr.io/meshcore-beacon/beacon-web:latest + image: ${BEACON_WEB_IMAGE:?Set BEACON_WEB_IMAGE to the reviewed web tag or digest} env_file: .env environment: - VITE_API_BASE=${VITE_API_BASE:?VITE_API_BASE environment variable is required} @@ -75,4 +75,4 @@ services: condition: service_healthy app: condition: service_started - restart: unless-stopped \ No newline at end of file + restart: unless-stopped From db0b31bdf6533075b77a9a08ccd1d86ea6cd2192 Mon Sep 17 00:00:00 2001 From: n30nex Date: Wed, 30 Sep 2026 14:46:37 -0400 Subject: [PATCH 41/69] docs: distinguish frozen release cut from later development --- RELEASE-CHECKLIST.md | 3 ++- ROADMAP.md | 2 ++ app_documentation/release-140-heads.json | 14 ++++++++++++-- app_documentation/release-140-preparation.md | 2 ++ 4 files changed, 18 insertions(+), 3 deletions(-) diff --git a/RELEASE-CHECKLIST.md b/RELEASE-CHECKLIST.md index d840e3b..147caf0 100644 --- a/RELEASE-CHECKLIST.md +++ b/RELEASE-CHECKLIST.md @@ -33,7 +33,8 @@ record](app_documentation/release-132-preparation.md) and dated evidence documen ## Alderson's release gates - [ ] Review/accept server #189, web #105 and docs #5. Verify CI on the actual - accepted commits, including changes after the recorded candidate. + accepted commits, including changes after the recorded candidate. Web #108/#109 + landed after the #107 cutoff and are not included in this preview. - [ ] Resolve the earlier 24h/3d raw-history requirement against upstream's restored 7d/30d controls and the conversion of old 3d observer links to 7d. - [ ] Verify desktop and physical iPhone Safari, English/French, keyboard, Back, diff --git a/ROADMAP.md b/ROADMAP.md index 9c464af..bc35b90 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,5 +1,7 @@ # Beacon parity and analytics roadmap +Later development: web #108/#109 landed after the validated #107 cutoff. They are not in preview 23945d59. Its own CI remains green and the PR is mergeable, but the current-base stack check flags the newer development head. Alderson must choose the release cutoff and validate any additions before tagging. + ## Beacon 1.4.0 candidate — 30 September **1.4.0 replaces the planned 1.3.2 release.** Alderson reviews and releases it, then switches `live.meshcore.ca` from CoreScope to production Beacon. `dev.meshcore.ca` stays online for development testing only, with separate application/data/configuration and matching development endpoints. Neither host was changed by this preparation. The server retains its independent version history. diff --git a/app_documentation/release-140-heads.json b/app_documentation/release-140-heads.json index 831421e..7b969a1 100644 --- a/app_documentation/release-140-heads.json +++ b/app_documentation/release-140-heads.json @@ -58,9 +58,19 @@ "Alderson review/acceptance and exact accepted-head CI", "24h/3d raw-history requirement versus upstream 7d/30d controls", "Physical iPhone Safari and production-host load/data validation", - "Stable image/tag publication and owner-controlled CoreScope cutover" + "Stable image/tag publication and owner-controlled CoreScope cutover", + "Decide whether to add web #108/#109 and validate the final accepted combination." ], "web_cutoff": "Accepted web #107 at b1f41dea; later changes require a release decision.", "web_validation_evidence": "release-140-final-20260930", - "server_validation_evidence": "release-140-20260930" + "server_validation_evidence": "release-140-20260930", + "later_development": { + "head": "61ab6697d9c727379e1de0e54aa18811a4b4bba6", + "prs": [ + 108, + 109 + ], + "included_in_preview": false, + "current_base_check": "refresh_required" + } } diff --git a/app_documentation/release-140-preparation.md b/app_documentation/release-140-preparation.md index 6565e11..9e702c4 100644 --- a/app_documentation/release-140-preparation.md +++ b/app_documentation/release-140-preparation.md @@ -1,5 +1,7 @@ # Beacon 1.4.0 release and CoreScope cutover +Later development: web #108/#109 landed after the validated #107 cutoff. They are not in preview 23945d59. Its own CI remains green and the PR is mergeable, but the current-base stack check flags the newer development head. Alderson must choose the release cutoff and validate any additions before tagging. + The planned Beacon/web release is **1.4.0**, replacing the earlier 1.3.2 proposal. Alderson reviews and releases the candidate, then controls the production switch. This document prepares that switch; it does not claim that either MeshCore Canada From a78abecba1b6c619a77d555b86c45ede79f7fb71 Mon Sep 17 00:00:00 2001 From: n30nex Date: Wed, 30 Sep 2026 17:22:16 -0400 Subject: [PATCH 42/69] docs: record combined 1.4.0 Pi candidate and telemetry recovery --- RELEASE-CHECKLIST.md | 8 +- ROADMAP.md | 16 ++-- app_documentation/release-140-heads.json | 67 ++++++++------- app_documentation/release-140-preparation.md | 85 ++++++++++++-------- 4 files changed, 96 insertions(+), 80 deletions(-) diff --git a/RELEASE-CHECKLIST.md b/RELEASE-CHECKLIST.md index 147caf0..508a1e3 100644 --- a/RELEASE-CHECKLIST.md +++ b/RELEASE-CHECKLIST.md @@ -23,7 +23,9 @@ record](app_documentation/release-132-preparation.md) and dated evidence documen - [x] Both deployment templates accept explicit production/development image references and matching URLs; missing image selection fails before deployment. - [x] Native server/PostgreSQL tests, restored-copy migration and bounded ingestion - replay pass. These checks do not establish production capacity. + replay pass, together with all 1,035 Windows/Pi web tests. Migration 045 + preserves raw counts and retained telemetry; removed partial rows are privately + backed up. These checks do not establish production capacity. - [x] Source/assets, public reads, fresh packets and 26 boundaries are verified on the review preview. Active revisions and the frontend receipt are in the [candidate manifest](app_documentation/release-140-heads.json). @@ -33,8 +35,8 @@ record](app_documentation/release-132-preparation.md) and dated evidence documen ## Alderson's release gates - [ ] Review/accept server #189, web #105 and docs #5. Verify CI on the actual - accepted commits, including changes after the recorded candidate. Web #108/#109 - landed after the #107 cutoff and are not included in this preview. + accepted commits, including changes after the recorded candidate. The preview + now includes web #108–#111 and server dev 14354b03, plus our release PRs. - [ ] Resolve the earlier 24h/3d raw-history requirement against upstream's restored 7d/30d controls and the conversion of old 3d observer links to 7d. - [ ] Verify desktop and physical iPhone Safari, English/French, keyboard, Back, diff --git a/ROADMAP.md b/ROADMAP.md index bc35b90..6616fd8 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,23 +1,21 @@ # Beacon parity and analytics roadmap -Later development: web #108/#109 landed after the validated #107 cutoff. They are not in preview 23945d59. Its own CI remains green and the PR is mergeable, but the current-base stack check flags the newer development head. Alderson must choose the release cutoff and validate any additions before tagging. +## Beacon 1.4.0 combined candidate — 30 September -## Beacon 1.4.0 candidate — 30 September +The requested refresh now includes Alderson's latest accepted server **14354b03** and web **e2d272e0**, plus our release PRs. The Pi review site runs **server 61b0322a / web 157525ef**, from server #189 and web #105. This supersedes the earlier #107 cutoff. Web #108–#111 provide the shared observer sidebar, labelled device details, unified packet observations and removal of the duplicate Observer page in Analytics. The server includes the partial-telemetry and counter-bucketing corrections. -**1.4.0 replaces the planned 1.3.2 release.** Alderson reviews and releases it, then switches `live.meshcore.ca` from CoreScope to production Beacon. `dev.meshcore.ca` stays online for development testing only, with separate application/data/configuration and matching development endpoints. Neither host was changed by this preparation. The server retains its independent version history. +**Beacon/web remains 1.4.0**, replacing the planned 1.3.2 release. My Atlas #97 is excluded until after 1.4.0 and still needs conflict resolution against the accepted release head. Alderson controls acceptance, stable tags and the production switch from CoreScope at `live.meshcore.ca`; `dev.meshcore.ca` remains development-only. Neither official host was changed. Server versions remain independent. -The review preview is server **9054acd8 / web 23945d59**, from server #189 and web #105. The frontend displays **1.4.0**. Atlas #97 remains excluded until after 1.4.0 and requires conflict resolution against the accepted release head. The active queues contain only these two release-preparation PRs; accepted parents are not restacked. The separate future Atlas manifest is `planning/review-stack-web-post140.json`. +Current-base checks and published-head CI pass. Native Go/PostgreSQL tests, restored-copy migration 045, the 3,200-input replay, and Windows/Pi web build/lint/**1,035 tests** pass. All **21 public assets**, both sources, **26 boundaries**, live packet delivery and desktop/French phone checks pass. The test-helper readiness race was fixed by draining probes through a unique marker; 100 repetitions of each affected test passed. Live ingestion behavior is unchanged by that helper fix. Existing image-tag and Compose receipts remain applicable to unchanged workflow/template content. -Published-head CI, native Go/PostgreSQL tests, restored-copy migration 044, the 3,200-input replay, and native web build/lint/**1029 tests** pass. Eight actual metadata-action tag cases and six Compose rendering cases pass. Development pushes cannot publish `latest`; stable release tags own it, and production deployment inputs must be pinned. All 20 public assets, both sources, 26 boundaries and desktop/phone English/French checks pass. The server update preserved the other 23 containers; the frontend preserved all 24. +Migration 045 removed **485 partial telemetry rows on the restored copy**, preserving raw counts, retained telemetry and node fingerprints. Immediately before live migration, **489 matching rows** were separately saved in the private checkpoint. The full restored/checksummed dump and this row export are retained on and off the Pi. -Backend recovery is `evidence/release-140-final-20260930/rollback.py`, checkpoint `release140-cutover-20260930T174802Z`, restoring 689bc232 / 00d859d9 while preserving new traffic and the compatible additive boundary table. It restores the frontend first if needed. Frontend-only recovery is `evidence/release-140-final-20260930/deploy-beacon-web.py rollback --evidence-dir release-140-final-20260930`, checkpoint `web-20260930T183247Z`. A restored/checksummed private dump is retained on and off the Pi. Configuration remains b3d96f52 / mode 0644; automatic zones, public admin/backup and foreign classification stay disabled. +Backend recovery is `evidence/sync-140-20260930/deploy-server.py rollback`, checkpoint `sync140-cutover-20260930T204913Z`. It restores **9054acd8 / 23945d59**, rolling back the new frontend first when needed. It preserves new traffic and the compatible telemetry cleanup; the private row export retains deleted rows for selective recovery. Frontend-only recovery is `evidence/sync-140-20260930/deploy-beacon-web.py rollback --evidence-dir sync-140-20260930`, checkpoint `web-20260930T211410Z`. Configuration remains b3d96f52 / mode 0644. The server update preserved the other 23 containers; the frontend preserved all 24. Automatic zones, public admin/backup and foreign classification stay disabled. -**Remaining owner gates:** review/acceptance, the 24h/3d raw-history requirement versus upstream 7d/30d controls, physical Safari, production-host capacity/data verification, approved Actions images/tags and the CoreScope route switch. The bounded replay and brief live sample do not certify production capacity. The earlier readiness-test race remains historical evidence. +**Remaining owner gates:** review/acceptance, the requested 24h/3d raw-history controls versus upstream 7d/30d controls, physical Safari, production-host capacity/data verification, approved Actions images/tags and the CoreScope switch. The bounded replay and 40-second live sample do not establish production capacity. [Current release/cutover plan](app_documentation/release-140-preparation.md) · [Exact candidate record](app_documentation/release-140-heads.json). -The final web cutoff includes #106/#107; later development is outside this recorded candidate until reviewed. - ## Historical 1.3.2 preparation ## 1.3.2 release preparation — 30 September diff --git a/app_documentation/release-140-heads.json b/app_documentation/release-140-heads.json index 7b969a1..12dfebb 100644 --- a/app_documentation/release-140-heads.json +++ b/app_documentation/release-140-heads.json @@ -1,6 +1,6 @@ { "release": "Beacon/web 1.4.0", - "verified_at": "2026-09-30T18:37:14.975711+00:00", + "verified_at": "2026-09-30T21:20:04.255270+00:00", "release_published": false, "production_switched": false, "roles": { @@ -10,24 +10,24 @@ }, "server": { "pr": 189, - "revision": "9054acd89f96fa3e117db954c384b778a33f7aad", - "base": "0e242574fb5ff852412c8c741877e104c59c4c6d", - "tree": "81335e7cdb2d9e19d81d547ffa2001e959f73782", - "binary_sha256": "870773fa64a8ce7179a1afda21e69ed830f331eaa56a75679bbadc45c1fb4fd9", - "source_sha256": "954bd52cebf9cd5024eb21400004e1e128c6a8e855ceb2d6f2f334eb32e6fba1", + "revision": "61b0322a5d2041987f154b7cea3870e848723890", + "base": "14354b03dc84fa5318914c8fc86444251f3f23cf", + "tree": "9d586f58e12bccef57f489e08c3a2ed5422d235e", + "binary_sha256": "d86c81b1821daa8726172701d4e5b86d8f1ee202fe364fe3d1c99dec9a2fd248", + "source_sha256": "93b0944a09a1bbdea0cd3c9f2d70ea7528feed6b0f20d99397528b5792434b41", "version_policy": "Independent server version; do not downgrade to web 1.4.0." }, "web": { "pr": 105, - "revision": "23945d5907b031345f8bc5e85b926532272d238b", - "base": "b1f41dea1d471a2d26941401cbe1b2d446995754", - "tree": "59293351133115673dd1770ba03cb4c5067482b6", + "revision": "157525ef3dd517e74cd40a404d5512ef14603845", + "base": "e2d272e08ee3cf043951f55f867375e3892d9bc8", + "tree": "cc73c740da440d1fa1d659a145d1a02eb741fc25", "version": "1.4.0", - "tests": 1029, - "source_sha256": "83b72731fc6c0dadf1b9cbd270a43fc9ccb2a66be869efb8caf7fce320344e38", - "dist_sha256": "b92255556988a538cd8961d44b3dfe53c9527de8e3d5bb4722cf40bfe547174b", - "index_sha256": "134093571a33bf3209ade58face8f38114a08f573fe8a4c6629d498a2df5172a", - "public_assets": 20 + "tests": 1035, + "source_sha256": "54ee7da23149401d829f2d1bc28f573ebe764423a401636764a5d507a0f9c004", + "dist_sha256": "95ad4ccb50ef05748211edfa39a9c8ca69bbae540b0f27e103482acd2560346c", + "index_sha256": "722254ca9585a751735d80eec5683baafe014e28dadb15460deba481d3686f8b", + "public_assets": 21 }, "atlas": { "pr": 97, @@ -38,39 +38,36 @@ }, "validation": { "native_postgres": true, - "restored_migration": "044_meshmapper_zones.sql", + "restored_migration": "045_clean_partial_observer_telemetry.sql", "replay_inputs": 3200, "replay_fixture_drops": 0, "image_tag_cases": 8, "compose_cases": 6, "public_boundaries": 26, - "windows_and_pi_web_tests": 1029 + "windows_and_pi_web_tests": 1035, + "restored_partial_rows_removed": 485, + "live_partial_rows_saved": 489, + "readiness_test_repetitions_each": 100, + "current_base_checks": true, + "published_head_ci": true }, "recovery": { - "server_checkpoint": "release140-cutover-20260930T174802Z", - "web_checkpoint": "web-20260930T183247Z", + "server_checkpoint": "sync140-cutover-20260930T204913Z", + "web_checkpoint": "web-20260930T211410Z", "config_sha256": "b3d96f52b25e38a0f3ecfb15cd2e492db1d4f494b18774598a4cfad295e2d62f", - "server_recipe": "evidence/release-140-final-20260930/rollback.py", - "web_recipe": "evidence/release-140-final-20260930/deploy-beacon-web.py rollback --evidence-dir release-140-final-20260930", - "frontend_only_restores": "79c09864bdf671ab19d203c07c60f6d23091c3d2 with server 9054acd8" + "server_recipe": "evidence/sync-140-20260930/deploy-server.py rollback", + "web_recipe": "evidence/sync-140-20260930/deploy-beacon-web.py rollback --evidence-dir sync-140-20260930", + "frontend_only_restores": "23945d5907b031345f8bc5e85b926532272d238b with server 61b0322a", + "backend_restores": "9054acd89f96fa3e117db954c384b778a33f7aad / 23945d5907b031345f8bc5e85b926532272d238b", + "data_policy": "Keep new traffic and compatible cleanup; private dump and 489-row export support selective recovery." }, "remaining": [ "Alderson review/acceptance and exact accepted-head CI", "24h/3d raw-history requirement versus upstream 7d/30d controls", "Physical iPhone Safari and production-host load/data validation", - "Stable image/tag publication and owner-controlled CoreScope cutover", - "Decide whether to add web #108/#109 and validate the final accepted combination." + "Stable image/tag publication and owner-controlled CoreScope cutover" ], - "web_cutoff": "Accepted web #107 at b1f41dea; later changes require a release decision.", - "web_validation_evidence": "release-140-final-20260930", - "server_validation_evidence": "release-140-20260930", - "later_development": { - "head": "61ab6697d9c727379e1de0e54aa18811a4b4bba6", - "prs": [ - 108, - 109 - ], - "included_in_preview": false, - "current_base_check": "refresh_required" - } + "web_cutoff": "Accepted dev e2d272e0 through #111, plus release PR #105. Later changes need review and validation.", + "web_validation_evidence": "sync-140-20260930", + "server_validation_evidence": "sync-140-20260930" } diff --git a/app_documentation/release-140-preparation.md b/app_documentation/release-140-preparation.md index 9e702c4..4b37a6e 100644 --- a/app_documentation/release-140-preparation.md +++ b/app_documentation/release-140-preparation.md @@ -1,7 +1,5 @@ # Beacon 1.4.0 release and CoreScope cutover -Later development: web #108/#109 landed after the validated #107 cutoff. They are not in preview 23945d59. Its own CI remains green and the PR is mergeable, but the current-base stack check flags the newer development head. Alderson must choose the release cutoff and validate any additions before tagging. - The planned Beacon/web release is **1.4.0**, replacing the earlier 1.3.2 proposal. Alderson reviews and releases the candidate, then controls the production switch. This document prepares that switch; it does not claim that either MeshCore Canada @@ -26,8 +24,8 @@ before starting a second stack; the Type 1 defaults assume one standalone stack. | Repository | Candidate | Scope | |---|---|---| -| Server | [#189](https://github.com/MeshCore-Beacon/beacon-server/pull/189), `9054acd89f96fa3e117db954c384b778a33f7aad`, based on `0e242574` | Stable image publishing and rollout documentation; includes the accepted Zones API importer and migration 044 | -| Web | [#105](https://github.com/MeshCore-Beacon/beacon-web/pull/105), `23945d5907b031345f8bc5e85b926532272d238b`, based on `b1f41dea` | Package/lock version 1.4.0, explicit deployment image selection and stable publishing; includes accepted Analytics #104 and phone/header/channel polish #106/#107 | +| Server | [#189](https://github.com/MeshCore-Beacon/beacon-server/pull/189), `61b0322a5d2041987f154b7cea3870e848723890`, based on `14354b03` | Stable image publishing and rollout documentation; includes accepted Zones API, partial-telemetry cleanup 045 and corrected counter buckets; drains readiness probes in the shared test helper | +| Web | [#105](https://github.com/MeshCore-Beacon/beacon-web/pull/105), `157525ef3dd517e74cd40a404d5512ef14603845`, based on `e2d272e0` | Package/lock version 1.4.0, explicit deployment image selection and stable publishing; includes accepted observer/sidebar/device and packet-analyzer changes through #111 | | Docs | [#5](https://github.com/MeshCore-Beacon/beacon-docs/pull/5) | Environment roles, pinned deployment inputs, release checklist, validation and recovery | | Web, deferred | [My Atlas #97](https://github.com/MeshCore-Beacon/beacon-web/pull/97) | Excluded until after 1.4.0; retains its single feature PR and saved-card design, but requires conflict resolution against current dev | @@ -44,10 +42,12 @@ join the production candidate. No Atlas changes belong in either release artifac ## Release notes draft - Observer monitoring combines packet activity, device telemetry and comparison, - with the observer list alongside the dashboard and an Observer view in Analytics. + with the shared observer sidebar and labelled device details. The duplicate + Observer page in Analytics is removed; comparison remains available there. - Packet, observer, node, route and map links provide retained evidence for investigation. Endpoint and path ambiguity remain visible rather than implying - a uniquely identified sender, relay or delivery path. + a uniquely identified sender, relay or delivery path. Packet analysis uses a + single observations list. - MeshMapper scope metadata and optional boundary imports improve regional views; configured fallbacks and missing/unknown scope states remain explicit. - Hourly analytics preserve summaries after raw packet expiry. Raw packet detail @@ -55,7 +55,8 @@ join the production candidate. No Atlas changes belong in either release artifac - English/French navigation, compact phone controls and on-demand packet maps improve presentation. Atlas remains outside this release. - Ingestion, route maintenance, cache invalidation, location reset handling and - deployment image separation include the accepted reliability/performance fixes. + partial-telemetry filtering, counter bucketing and deployment image separation + include the accepted reliability/performance fixes. Production capacity still needs validation on its intended host. ## Image publishing and configuration @@ -115,7 +116,10 @@ history. The historical counter discrepancy is not resolved by changing products The agreed candidate policy remains **72-hour raw packets, 30-day hourly summaries and 720-hour telemetry**. Migration 043 clears stale zero/omitted advert positions; -044 adds separate imported zone-boundary storage. Test migrations on a restored +044 adds separate imported zone-boundary storage; 045 deletes partial telemetry +rows where noise floor and both airtime counters are zero. Preserve those rows in +a private export before migrating. The code skips uptime-only radio-stat updates +and ignores invalid counter dips/replays/spikes. Test migrations on a restored copy before applying them to the production Beacon database. Verify raw counts, retained summaries and unaffected node locations. Keep the pre-change dump, application images, configuration and proxy route available for rollback. @@ -159,34 +163,49 @@ public backup and foreign-node classification are not enabled by this release pr ## Validation and remaining release decisions The exact PR heads are deployed at the review site and its footer shows 1.4.0. -The web cutoff is accepted #107 at `b1f41dea`; the refreshed web #105 is -`23945d5907b031345f8bc5e85b926532272d238b`. Windows and Pi tests match. -Published-head CI, the native PostgreSQL suite (including zone-boundary storage), -restored migration 044 and the 3,200-input replay pass. The replay retained 100 -packets / 800 observations / 100 decrypted messages and the expected 800 ordinary -and 2,400 opted-in events, with zero fixture drops. Native frontend build/lint and -all **1029 tests** pass. Eight tag-generation cases, six configuration-rendering -cases, public asset/source/boundary checks and desktop/phone English/French browser -checks pass. See the [exact candidate record](release-140-heads.json). - -The current private checkpoint is `release140-cutover-20260930T174802Z`. -The combined Pi recovery is `python3 evidence/release-140-final-20260930/rollback.py`. -It first restores frontend 79c09864, then invokes the guarded backend recovery to -restore server 689bc232 / web 00d859d9. New traffic and the compatible additive -044 table are preserved. Do not skip the newest frontend rollback when using -an older phase's recovery script. - -Frontend-only recovery is `python3 evidence/release-140-final-20260930/deploy-beacon-web.py rollback --evidence-dir release-140-final-20260930`, checkpoint `web-20260930T183247Z`; -it keeps server 9054acd8 and restores frontend 79c09864. The private dump was -restored and checksum-verified on and off the Pi. The server update left 23 other -containers unchanged; each frontend publication left all 24 unchanged. +The refreshed pair is server **61b0322a** on accepted dev **14354b03**, and web +**157525ef** on accepted dev **e2d272e0** through #111. This replaces the earlier +#107 cutoff at the contributor's request. Current-base checks and published-head +CI pass; web CodeQL remains skipped under existing policy. + +Native PostgreSQL tests, restored-copy migration 045 and the 3,200-input replay +pass. The restored copy removed 485 partial telemetry rows while preserving raw +counts, retained telemetry and node fingerprints. All 489 matching live rows were +saved separately before migration. The replay retained 100 packets / 800 +observations / 100 decrypted messages and 800 ordinary / 2,400 opted-in events, +with zero fixture drops in 12.51 seconds. The readiness-test race is fixed in the +shared helper and both affected tests passed 100 repetitions. + +Windows and Pi frontend build/lint and all **1,035 tests** pass. Prior eight +tag-generation and six configuration-rendering receipts apply to unchanged +workflow/template content. All 21 public assets, source archives and 26 boundaries +match. Desktop/French phone browser checks and live delivery pass. The 40-second +runtime sample had fresh traffic and no recorded queue overflow, SQL error, +fallback, reconnect or panic; it does not establish production capacity. +See the [exact candidate record](release-140-heads.json). + +The private checkpoint is `sync140-cutover-20260930T204913Z`, restored and +checksum-verified on and off the Pi. The matching partial-row export is retained +alongside it. Combined recovery is +`python3 evidence/sync-140-20260930/deploy-server.py rollback`. +It first restores the previous frontend when necessary, then restores server +**9054acd8 / web 23945d59**. New traffic and the compatible telemetry cleanup stay +in place; use the private row export for selective recovery if needed. Do not +overwrite newly received traffic with the old full database dump. + +Frontend-only recovery is +`python3 evidence/sync-140-20260930/deploy-beacon-web.py rollback --evidence-dir sync-140-20260930`, +checkpoint `web-20260930T211410Z`. It keeps server 61b0322a and restores web +23945d59. The backend update left 23 other containers unchanged; frontend +publication left all 24 unchanged. Configuration and retention were preserved. The current upstream UI has 24h/7d/30d controls, including raw-evidence views, and old 3d observer links select 7d. This conflicts with the earlier requested 24h/3d raw-history controls. Alderson must resolve that requirement or explicitly accept the changed behavior before release; the version change does not resolve it. -The existing test readiness-message race, any native validation retries and replay -timing limits remain recorded. Production-host capacity, the actual CoreScope -cutover and physical Safari validation stay explicit owner acceptance steps. -The earlier [1.3.2 preparation record](release-132-preparation.md) is historical. +Production-host capacity, the actual CoreScope cutover and physical Safari +validation remain owner acceptance steps. No official host, stable tag or upstream +merge was changed by this refresh. The earlier +[1.3.2 preparation record](release-132-preparation.md) and dated receipts remain +historical; use the current recovery commands for this deployed pair. From d91eec9afc7994b28601f790233e425c919717de Mon Sep 17 00:00:00 2001 From: n30nex Date: Wed, 30 Sep 2026 18:39:28 -0400 Subject: [PATCH 43/69] docs: sequence post-1.4 parity work and activate route consistency phase --- ROADMAP.md | 28 ++++++---- app_documentation/post-140-roadmap.md | 76 +++++++++++++++++++++++++++ 2 files changed, 95 insertions(+), 9 deletions(-) create mode 100644 app_documentation/post-140-roadmap.md diff --git a/ROADMAP.md b/ROADMAP.md index 6616fd8..2a1d946 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,5 +1,17 @@ # Beacon parity and analytics roadmap +## Active post-1.4 phased work — 30 September + +The contributor approved proceeding toward and beyond CoreScope feature parity. +The sequence is **saved-route correctness (#183) → My Atlas (#97, after 1.4.0) +→ node/trace dashboards and investigation → search/saved views/channel analytics +→ topology/distance/hash tools → bounded replay/reach/timing → regional crossing evidence**. +Phase 1 is active. Each phase ends in focused reviewable PRs, exact Pi validation, +updated source/changelog and retained rollback; release numbers remain a maintainer decision. +The current 1.4.0 handoff remains recorded below, with Atlas excluded. + +[Phases, contracts and acceptance](). + ## Beacon 1.4.0 combined candidate — 30 September The requested refresh now includes Alderson's latest accepted server **14354b03** and web **e2d272e0**, plus our release PRs. The Pi review site runs **server 61b0322a / web 157525ef**, from server #189 and web #105. This supersedes the earlier #107 cutoff. Web #108–#111 provide the shared observer sidebar, labelled device details, unified packet observations and removal of the duplicate Observer page in Analytics. The server includes the partial-telemetry and counter-bucketing corrections. @@ -242,20 +254,18 @@ The September 24 consolidation check built accepted server `c02317a4` and retain ## Next phases -The September 20 #116 investigation has a new [current-build result](https://github.com/MeshCore-Beacon/beacon-server/issues/116#issuecomment-5753626821): a 600-second unmodified Pi capture kept both feeds connected and retained 2,169 new observations, with no ping timeout, disconnect, deadline, SQLSTATE error or HTTP 5xx response. App/PostgreSQL CPU averaged 2.14%/3.96% of one core. The preceding 3h39 log likewise has no MQTT loss or deadline error. Timestamp warnings were classified separately. This did not measure callback or pool-acquisition duration and does not establish the original cause or production capacity. No application, ordering, acknowledgement or service change was made; #116 remains open. Further capture should follow a recurrence or meaningful workload change, rather than repeatedly sampling the same healthy state. - -1. **Reviews and listed issues first.** The #181 correction is submitted as #184. Next address [server #183](https://github.com/MeshCore-Beacon/beacon-server/issues/183), preserving honest saved-route evidence when prefix widths change. Recheck maintainer feedback first; broad partial issues remain open. -2. **Accept the current queue in dependency order.** Server #167 -> #169 -> #172 -> #174 -> #176; independent #166. Web #75 -> #79 -> #80 -> #81 -> #83 -> #85 -> #87 -> #89 -> #92 -> #95. Independent server #184 follows dev. Use the published-head table in the integration record. Maintainers choose merges, the release breakpoint, versions, tags and main promotion. -3. **Optional MeshMapper boundaries.** Scope import and channel tags are already implemented in #174/#176/#89. Next use the [boundary plan](app_documentation/meshmapper-boundaries-plan.md), preserving manual boundary priority, cached valid geometry and separate scope/forwarding evidence. Crossing analytics remain a later focused slice. -4. **Connected investigation and presentation.** Packet/route/observer links and return navigation are delivered for review. Continue node/trace presentation, distinct analytics questions and quality of life under the approved observer plan; address #99/#12 where the work overlaps. Full parity is not yet claimed. +Use the [active post-1.4 sequence](app_documentation/post-140-roadmap.md). The +older dependency queue and optional-boundary implementation are delivered in the +current candidate and must not be scheduled again. The next focused implementation +is server #183; Atlas follows separately after 1.4.0. Scope/route crossing analysis, +node/trace presentation and new analytics retain their own acceptance contracts. ## Listed work still open | Issue | Remaining scope | |---|---| -| [Server #181](https://github.com/MeshCore-Beacon/beacon-server/issues/181) | Fixed by #184; awaits maintainer acceptance | -| [Server #183](https://github.com/MeshCore-Beacon/beacon-server/issues/183) | Saved-route hash-prefix metadata can lag a change of width; next focused issue | -| [Web #94](https://github.com/MeshCore-Beacon/beacon-web/issues/94) | Corrected periods/labels in #95; awaits acceptance | +| [Server #183](https://github.com/MeshCore-Beacon/beacon-server/issues/183) | Active first post-1.4 phase: saved-route metadata and evidence consistency across width changes | +| [Web #94](https://github.com/MeshCore-Beacon/beacon-web/issues/94) | The 24h/3d correction landed through #95/#99, then upstream restored 7d/30d; the retention-window decision remains open | | [Server #116](https://github.com/MeshCore-Beacon/beacon-server/issues/116) | #179/#180 are integrated and pass bounded replay/route-lock checks; attributing the historical incident still requires matching evidence | | [Server #99](https://github.com/MeshCore-Beacon/beacon-server/issues/99) | Advert names and ACK/TRACE/PING references are accepted; define any remaining packet-type formats | | [Server #60](https://github.com/MeshCore-Beacon/beacon-server/issues/60) | Remaining administration/worker/persistence behavior; account records do not establish login sessions | diff --git a/app_documentation/post-140-roadmap.md b/app_documentation/post-140-roadmap.md new file mode 100644 index 0000000..7f69f21 --- /dev/null +++ b/app_documentation/post-140-roadmap.md @@ -0,0 +1,76 @@ +# Beacon after 1.4: parity and further development + +Approved direction, 30 September 2026: deliver useful CoreScope feature parity in +small validated phases, then extend Beacon's regional and evidence-based analysis. +Version assignments remain with maintainers. The 1.4.0 release handoff is separate +from this development queue; My Atlas remains excluded from that release. + +## Delivery order + +| Phase | Deliverable | Completion evidence | Status | +|---|---|---|---| +| 1 — correctness suitable for 1.4.x | Saved-route hash-width consistency, then retention-aware time controls and remaining French/mobile/accessibility fixes | Route identity survives representation changes; evidence pagination and shared windows do not silently change path; raw and summary periods match available data | **Active: server #183** | +| 2 — first feature after 1.4.0 | My Atlas saved-node monitoring | Refresh single web PR #97; saved identities/order survive; compact cards, expandable Heard by/statistics and existing entity links work in English/French on desktop/phone | Held until the release; conflict refresh required | +| 3 — node and trace investigation | Node dashboard, activity/type/signal/hop analysis, trace reception timeline and complete return navigation | Separate attributed node traffic from possible prefix matches; packet → route → node/observer → map links preserve selection, filters and Back | Queued | +| 4 — find and compare | Bounded global entity search, saved views/filters, Atlas-node filters, channel activity and hearing context | Search/paging/share links agree; channel key/history availability is explicit; comparisons use aligned windows | Queued | +| 5 — network structure | Observed route segments/alternatives, topology, distance, hash ambiguity and prefix/path inspection | Count evidence at the correct grain; separate observed ambiguity from static conflicts; use valid coordinates and show unresolved hops | Queued | +| 6 — history and reach | Bounded retained map replay, observer reach and timing analysis, comparable fleet telemetry | Replay preserves ordering and retention limits; confirmed identities are separate from unresolved prefixes; timing/counter gaps are not labelled packet loss | Queued | +| Beyond parity | Regional boundary/scope crossing investigation and links between observations, discovered scopes and route changes | Each relationship links to retained evidence; distinguish reported locations from inferred paths and scope names from geography | Queued after supporting phases | + +The initial Atlas cards show a bounded sample of the latest 200 retained +origin-key reports per node. Complete node totals and longer history need an +explicit aggregate contract in phase 3; do not relabel the sample as total traffic. +Browser-local cards remain the first release. Account sync or MeshMapper login +needs a separately agreed authentication and ownership contract. + +## First implementation: saved-route evidence + +[Server #183](https://github.com/MeshCore-Beacon/beacon-server/issues/183) remains +open. The same fully resolved node chain can arrive with different hash widths, +while its saved prefix metadata currently stays at the first representation. +That can omit newer exact-path observations from route evidence. + +Keep the stable IATA/node-chain identity. Define how the latest representation is +updated and how an already-open evidence page retains its selected representation. +Cover successive 1/2/3-byte paths, unchanged repeats, malformed/missing metadata, +ambiguous identities, cursor precision and shared-window behavior. Preserve exact +bytes and the existing bounded observation index; do not replace the query with +a broad short-prefix or raw-history scan. Avoid a schema change unless the agreed +behavior requires one. This phase must include a real PostgreSQL regression and +the exact combined Pi build before its preview is called verified. + +## Parallel priorities, scheduled deliberately + +- Operational work: remaining admin configuration scope, browser access and + import/restore, deployment-file coverage and remote/scheduled backup under + [server #60](https://github.com/MeshCore-Beacon/beacon-server/issues/60) and + [#72](https://github.com/MeshCore-Beacon/beacon-server/issues/72). +- MQTT timeout attribution under + [#116](https://github.com/MeshCore-Beacon/beacon-server/issues/116): collect + evidence on recurrence or a meaningful workload change. A healthy short sample + does not identify the historical cause. +- Complete translations under + [web #12](https://github.com/MeshCore-Beacon/beacon-web/issues/12), with touched + interface text translated in the same feature PR. +- Reconcile older open tickets against accepted code. Several observer, packet + and route tickets were implemented through the consolidation batch; their open + state alone is not a new feature gap. Close only after their full scope is checked. + +## Shared acceptance criteria + +Every new view defines its counting unit, effective period, available history, +sample size, missing data and ambiguity. Raw packets and observations cannot be +reconstructed after expiry; longer-lived summaries must state what they preserve. +Reuse the existing observer, packet, route, map and chart components. Load only the +selected data, coalesce updates, bound requests and test query plans on +representative data. New aggregation is justified by a measured query need. + +Validate source and actual running artifacts, native PostgreSQL behavior, relevant +ingestion/reconnect regressions, desktop/phone, English/French, keyboard and shared +links. Publish matching source/changelog and preserve rollback without overwriting +new traffic. Compare CPU/memory/storage and latency against the preceding build; +a fixture or short Pi sample does not certify production capacity. + +Maintainers control merges, version numbers, stable artifacts and the production +switch. `live.meshcore.ca` is the planned production destination; `dev.meshcore.ca` +remains development-only. The Canadaverse Pi remains the contribution preview. From bb79e1f077b19c23f7c7710b29771fc5317a36c4 Mon Sep 17 00:00:00 2001 From: n30nex Date: Wed, 30 Sep 2026 18:59:31 -0400 Subject: [PATCH 44/69] docs: isolate n30nex-test work and define pinned route evidence --- ROADMAP.md | 2 +- app_documentation/post-140-roadmap.md | 6 ++++-- app_documentation/saved-route-evidence.md | 8 +++++--- 3 files changed, 10 insertions(+), 6 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index 2a1d946..2251356 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -6,7 +6,7 @@ The contributor approved proceeding toward and beyond CoreScope feature parity. The sequence is **saved-route correctness (#183) → My Atlas (#97, after 1.4.0) → node/trace dashboards and investigation → search/saved views/channel analytics → topology/distance/hash tools → bounded replay/reach/timing → regional crossing evidence**. -Phase 1 is active. Each phase ends in focused reviewable PRs, exact Pi validation, +Phase 1 is active on the requested `n30nex-test` branches in server, web and docs. Each phase ends in focused reviewable PRs, exact Pi validation, updated source/changelog and retained rollback; release numbers remain a maintainer decision. The current 1.4.0 handoff remains recorded below, with Atlas excluded. diff --git a/app_documentation/post-140-roadmap.md b/app_documentation/post-140-roadmap.md index 7f69f21..a67f364 100644 --- a/app_documentation/post-140-roadmap.md +++ b/app_documentation/post-140-roadmap.md @@ -2,6 +2,8 @@ Approved direction, 30 September 2026: deliver useful CoreScope feature parity in small validated phases, then extend Beacon's regional and evidence-based analysis. +Post-1.4 work lives on the explicitly requested `n30nex-test` branches in server, +web and docs, with focused pull requests and the Pi preview as validation. Version assignments remain with maintainers. The 1.4.0 release handoff is separate from this development queue; My Atlas remains excluded from that release. @@ -9,7 +11,7 @@ from this development queue; My Atlas remains excluded from that release. | Phase | Deliverable | Completion evidence | Status | |---|---|---|---| -| 1 — correctness suitable for 1.4.x | Saved-route hash-width consistency, then retention-aware time controls and remaining French/mobile/accessibility fixes | Route identity survives representation changes; evidence pagination and shared windows do not silently change path; raw and summary periods match available data | **Active: server #183** | +| 1 — correctness suitable for 1.4.x | Saved-route hash-width consistency, then retention-aware time controls and remaining French/mobile/accessibility fixes | Route identity survives representation changes; evidence pagination and shared windows do not silently change path; raw and summary periods match available data | **Active: server #183 / PR #192** | | 2 — first feature after 1.4.0 | My Atlas saved-node monitoring | Refresh single web PR #97; saved identities/order survive; compact cards, expandable Heard by/statistics and existing entity links work in English/French on desktop/phone | Held until the release; conflict refresh required | | 3 — node and trace investigation | Node dashboard, activity/type/signal/hop analysis, trace reception timeline and complete return navigation | Separate attributed node traffic from possible prefix matches; packet → route → node/observer → map links preserve selection, filters and Back | Queued | | 4 — find and compare | Bounded global entity search, saved views/filters, Atlas-node filters, channel activity and hearing context | Search/paging/share links agree; channel key/history availability is explicit; comparisons use aligned windows | Queued | @@ -26,7 +28,7 @@ needs a separately agreed authentication and ownership contract. ## First implementation: saved-route evidence [Server #183](https://github.com/MeshCore-Beacon/beacon-server/issues/183) remains -open. The same fully resolved node chain can arrive with different hash widths, +open, with the server correction in [PR #192](https://github.com/MeshCore-Beacon/beacon-server/pull/192). The same fully resolved node chain can arrive with different hash widths, while its saved prefix metadata currently stays at the first representation. That can omit newer exact-path observations from route evidence. diff --git a/app_documentation/saved-route-evidence.md b/app_documentation/saved-route-evidence.md index 282083b..e94988f 100644 --- a/app_documentation/saved-route-evidence.md +++ b/app_documentation/saved-route-evidence.md @@ -2,10 +2,12 @@ Known-route responses include `pathKey`, a stable identity within the route's IATA. Use `GET /api/v1/routes/{iata}/{pathKey}/observations` to fetch the **full saved route** and retained report references. Search results can contain a subsegment while sharing the full route's key; the evidence response always describes the complete saved sequence. -The match requires the complete saved `pathBytes`, `hashSize`, hop count and IATA. A compact digest index narrows candidates, but full bytes are still compared. Other hash widths, TRACE readings/intended routes and unclassified legacy observations are excluded. Matching short prefixes does not confirm historical node identities, forwarding or delivery. The stored route counter can include repeated processing and outlive raw reports; it is not a retained-result total. +New route processing updates the current `hash_prefix` while retaining the IATA/node-chain identity, first-seen time and accumulated counter. The default evidence request uses that latest processed representation. It is not a history of when a radio changed its configuration. -The web interface offers **24h / 3d** and caps raw route/comparison selections at 72 hours to match the preview's packet retention. The API retains its bounded 30-day maximum for deployments with longer retention; it cannot recover expired reports. +The match requires one complete `pathBytes`, `hashSize`, hop count and IATA. A compact digest index narrows candidates, but full bytes are still compared. Other hash widths, TRACE readings/intended routes and unclassified legacy observations are excluded. Matching short prefixes does not confirm historical node identities, forwarding or delivery. The stored route counter can include repeated processing and outlive raw reports; it is not a retained-result total. -`range` defaults to `24h` and accepts durations up to `720h`, anchored on the server. Alternatively supply both `since` and exclusive `until` in epoch milliseconds, with a maximum 30-day span and no future end. `limit` defaults to 50 and is capped at 200. Follow `nextPageCursor` as `pageCursor`; its route, window and microsecond/ID boundary are pinned. Do not combine it with another range, or change its explicit window. Numeric legacy `cursor` is unsupported. Responses include effective window bounds, `matchAvailable`, an empty `items` array when no matching raw evidence remains, and `hasMore`; no total-count scan or packet/message body is added. Malformed saved path metadata is explicitly unavailable, and missing routes return 404. +The API retains its bounded 30-day maximum for deployments with longer retention; it cannot recover expired reports. The current upstream interface offers 24h/7d/30d; the requested 24h/3d raw-history policy remains a separate release decision. + +`range` defaults to `24h` and accepts durations up to `720h`, anchored on the server. Alternatively supply both `since` and exclusive `until` in epoch milliseconds, with a maximum 30-day span and no future end. `limit` defaults to 50 and is capped at 200. Follow `nextPageCursor` as `pageCursor`; new v2 cursors pin route, window, exact width/path bytes and the microsecond/ID boundary. The server still accepts legacy v1 cursors, which have no representation pin. Provide `hashSize` and `pathBytes` together to reopen the exact representation from a prior response; the new web Copy link action includes these with the fixed time window. Older links without them still select the current representation. Conflicting explicit selectors and cursor pins are rejected. A pin with another width must match each saved node identity's public-key prefix, using the existing batched node read; a missing identity or different chain is rejected rather than substituted. The response's displayed hops use the selected bytes. This remains an exact prefix match, not proof of historical forwarding by those nodes. Do not combine it with another range, or change its explicit window. Numeric legacy `cursor` is unsupported. Responses include effective window bounds, `matchAvailable`, an empty `items` array when no matching raw evidence remains, and `hasMore`; no total-count scan or packet/message body is added. Malformed saved path metadata is explicitly unavailable, and missing routes return 404. Migration 041 builds the compact observation index concurrently. Keep it as a single statement outside a transaction; the existing runner handles an interrupted or already-built index before recording completion. It does not alter retained rows or expiry configuration. Native PostgreSQL tests cover ties below millisecond precision, cursor scope, different widths/sites, TRACE/unknown exclusions, raw expiry, index retry and custom/generic indexed plans. From 8cfdcf6bbc31b469897f52e20808ae78a34d19de Mon Sep 17 00:00:00 2001 From: n30nex Date: Wed, 30 Sep 2026 19:16:57 -0400 Subject: [PATCH 45/69] docs: include Atlas in experiment and record daily upstream checks --- ROADMAP.md | 4 ++-- app_documentation/post-140-roadmap.md | 13 +++++++++---- 2 files changed, 11 insertions(+), 6 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index 2251356..da2bd58 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -3,10 +3,10 @@ ## Active post-1.4 phased work — 30 September The contributor approved proceeding toward and beyond CoreScope feature parity. -The sequence is **saved-route correctness (#183) → My Atlas (#97, after 1.4.0) +The sequence is **saved-route correctness (#183) → My Atlas (#97, experimental now; release after 1.4.0) → node/trace dashboards and investigation → search/saved views/channel analytics → topology/distance/hash tools → bounded replay/reach/timing → regional crossing evidence**. -Phase 1 is active on the requested `n30nex-test` branches in server, web and docs. Each phase ends in focused reviewable PRs, exact Pi validation, +Phases 1 and 2 are combined on the requested `n30nex-test` branches in server, web and docs. The Pi preview will run this experimental composition, with daily upstream dev/main compatibility checks and no review pings. Each phase ends in focused reviewable PRs, exact Pi validation, updated source/changelog and retained rollback; release numbers remain a maintainer decision. The current 1.4.0 handoff remains recorded below, with Atlas excluded. diff --git a/app_documentation/post-140-roadmap.md b/app_documentation/post-140-roadmap.md index a67f364..7cc03ea 100644 --- a/app_documentation/post-140-roadmap.md +++ b/app_documentation/post-140-roadmap.md @@ -4,15 +4,20 @@ Approved direction, 30 September 2026: deliver useful CoreScope feature parity i small validated phases, then extend Beacon's regional and evidence-based analysis. Post-1.4 work lives on the explicitly requested `n30nex-test` branches in server, web and docs, with focused pull requests and the Pi preview as validation. -Version assignments remain with maintainers. The 1.4.0 release handoff is separate -from this development queue; My Atlas remains excluded from that release. +The branch is experimental: keep existing PRs in draft and do not request or ping +for review until the contributor asks. Daily upstream dev/main compatibility checks +are scheduled; inspect dirty work first, use isolated trial merges, and report only +new changes, conflicts or required decisions. Routine checks do not push, deploy +or merge changes automatically. Version assignments remain with maintainers. The 1.4.0 release handoff is separate +from this development queue; My Atlas remains excluded from that release, but is explicitly included in the +experimental branch and Pi preview at the contributor's request. ## Delivery order | Phase | Deliverable | Completion evidence | Status | |---|---|---|---| -| 1 — correctness suitable for 1.4.x | Saved-route hash-width consistency, then retention-aware time controls and remaining French/mobile/accessibility fixes | Route identity survives representation changes; evidence pagination and shared windows do not silently change path; raw and summary periods match available data | **Active: server #183 / PR #192** | -| 2 — first feature after 1.4.0 | My Atlas saved-node monitoring | Refresh single web PR #97; saved identities/order survive; compact cards, expandable Heard by/statistics and existing entity links work in English/French on desktop/phone | Held until the release; conflict refresh required | +| 1 — correctness suitable for 1.4.x | Saved-route hash-width consistency, then retention-aware time controls and remaining French/mobile/accessibility fixes | Route identity survives representation changes; evidence pagination and shared windows do not silently change path; raw and summary periods match available data | Implemented in experimental server PR #192 and the web branch; final Pi checks in progress | +| 2 — first feature after 1.4.0 | My Atlas saved-node monitoring | Carry the feature from web PR #97 into the experiment; saved identities/order survive; compact cards, expandable Heard by/statistics and existing entity links work in English/French on desktop/phone | Integrated into n30nex-test from the two feature commits; combined validation in progress | | 3 — node and trace investigation | Node dashboard, activity/type/signal/hop analysis, trace reception timeline and complete return navigation | Separate attributed node traffic from possible prefix matches; packet → route → node/observer → map links preserve selection, filters and Back | Queued | | 4 — find and compare | Bounded global entity search, saved views/filters, Atlas-node filters, channel activity and hearing context | Search/paging/share links agree; channel key/history availability is explicit; comparisons use aligned windows | Queued | | 5 — network structure | Observed route segments/alternatives, topology, distance, hash ambiguity and prefix/path inspection | Count evidence at the correct grain; separate observed ambiguity from static conflicts; use valid coordinates and show unresolved hops | Queued | From 0279c020f3fb26f5f9ea6042e38409eec5221a26 Mon Sep 17 00:00:00 2001 From: n30nex Date: Wed, 30 Sep 2026 19:34:36 -0400 Subject: [PATCH 46/69] docs: record validated experimental Atlas and route preview --- ROADMAP.md | 8 +-- app_documentation/n30nex-test-preview.md | 80 ++++++++++++++++++++++++ app_documentation/post-140-roadmap.md | 6 +- 3 files changed, 88 insertions(+), 6 deletions(-) create mode 100644 app_documentation/n30nex-test-preview.md diff --git a/ROADMAP.md b/ROADMAP.md index da2bd58..6803994 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -6,15 +6,15 @@ The contributor approved proceeding toward and beyond CoreScope feature parity. The sequence is **saved-route correctness (#183) → My Atlas (#97, experimental now; release after 1.4.0) → node/trace dashboards and investigation → search/saved views/channel analytics → topology/distance/hash tools → bounded replay/reach/timing → regional crossing evidence**. -Phases 1 and 2 are combined on the requested `n30nex-test` branches in server, web and docs. The Pi preview will run this experimental composition, with daily upstream dev/main compatibility checks and no review pings. Each phase ends in focused reviewable PRs, exact Pi validation, +Phases 1 and 2 are combined on the requested `n30nex-test` branches in server, web and docs. The Pi preview runs this experimental composition, with daily upstream dev/main compatibility checks and no review pings. The route fix and Atlas are validated; node/trace dashboards and deeper investigation are next. Each phase ends in focused reviewable PRs, exact Pi validation, updated source/changelog and retained rollback; release numbers remain a maintainer decision. The current 1.4.0 handoff remains recorded below, with Atlas excluded. -[Phases, contracts and acceptance](). +[Phases, contracts and acceptance](app_documentation/post-140-roadmap.md) · [Current experimental preview](app_documentation/n30nex-test-preview.md). -## Beacon 1.4.0 combined candidate — 30 September +## Recorded 1.4.0 checkpoint — superseded on the Pi by the experiment -The requested refresh now includes Alderson's latest accepted server **14354b03** and web **e2d272e0**, plus our release PRs. The Pi review site runs **server 61b0322a / web 157525ef**, from server #189 and web #105. This supersedes the earlier #107 cutoff. Web #108–#111 provide the shared observer sidebar, labelled device details, unified packet observations and removal of the duplicate Observer page in Analytics. The server includes the partial-telemetry and counter-bucketing corrections. +The requested refresh now includes Alderson's latest accepted server **14354b03** and web **e2d272e0**, plus our release PRs. At that checkpoint, the Pi review site ran **server 61b0322a / web 157525ef**, from server #189 and web #105. This supersedes the earlier #107 cutoff. Web #108–#111 provide the shared observer sidebar, labelled device details, unified packet observations and removal of the duplicate Observer page in Analytics. The server includes the partial-telemetry and counter-bucketing corrections. **Beacon/web remains 1.4.0**, replacing the planned 1.3.2 release. My Atlas #97 is excluded until after 1.4.0 and still needs conflict resolution against the accepted release head. Alderson controls acceptance, stable tags and the production switch from CoreScope at `live.meshcore.ca`; `dev.meshcore.ca` remains development-only. Neither official host was changed. Server versions remain independent. diff --git a/app_documentation/n30nex-test-preview.md b/app_documentation/n30nex-test-preview.md new file mode 100644 index 0000000..e554aab --- /dev/null +++ b/app_documentation/n30nex-test-preview.md @@ -0,0 +1,80 @@ +# Experimental n30nex-test preview + +Verified 30 September 2026. The contributor requested separate `n30nex-test` +branches, Atlas in the Pi preview, and daily checks against upstream dev/main. +This work is experimental. Existing server #192 and docs #7 remain drafts; do not +request reviews, mention reviewers or promote the work until asked. The web branch +has no new pull request. The stable 1.4.0 release handoff remains separate. + +[Open My Atlas](https://canadaverse.org/beacon-dev/?tab=MyAtlas) · +[Corresponding source and changelog](https://canadaverse.org/beacon-dev/source.html) + +| Repository | Experimental branch | Validated/deployed source | +|---|---|---| +| Server | `n30nex-test`, `9505ad444268c552e08e46613ed8f2e0f8fd78a8` | Same revision; includes upstream dev `5b177d6f`, release parent #189 and the route-prefix fix in draft #192 | +| Web | `n30nex-test`, `dc3268f7468bd9b3ae280243411c231fde6dd6ae` | Built as `b99b6e779d7bc4ec47e0e5202191d6aac9510ee0`; both have exactly the same `29d2513fe6f57f2aada079977e8eed40d72bc1dd` source tree | +| Docs | `n30nex-test`, draft #7 | Active phased roadmap, experiment contract and this record | + +## Included work + +- Saved routes update the current processed prefix representation without changing + their stable IATA/node-chain identity. New cursors and copied links pin one exact + width/path and time window; unrelated bytes or missing identities cannot silently + select another chain. The existing indexed observation query and schema are reused. +- My Atlas includes the two feature commits from web #97, adapted to the current + upstream navigation and translations. Existing tab order stays intact, with Atlas + beside Observers. Saved full-key cards, activity bars, signal meters, expandable + Heard by and statistics work in English and French. The cards remain bounded + samples of up to 200 origin-key reports, not complete node totals. +- The ingestion status-text and logging corrections from upstream `5b177d6f` are + included. Public admin/backup, automatic zones and foreign classification remain + disabled; saved configuration and retention are unchanged. + +## Compatibility and validation + +Trial merges are clean for server and web against upstream dev and main, and for +docs against main. Web main `5ac36ce` was squash-created from `6daf342c` in release +PR #27. Both have the identical `03ab89640d90d77bc844b59b4945c4ad899f9cf0` tree; +the original is already an ancestor of current dev. A source-preserving ancestry +merge records that fact and removes the otherwise inherited 45 merge conflicts. +No source file was replaced during that reconciliation. The deployed web build +keeps its actual b99b6e7 identity and matching source archive; it is not relabelled. + +Native Go/PostgreSQL tests and server CI pass. The route regression changes widths +between pages and checks shared links, repeats, malformed selectors, missing and +colliding identities, and exact indexed plans. The 200,000-row fixture used the +existing index under custom and generic plans (0.097/0.090 ms execution in this +fixture). The 3,200-input replay retained the expected 100 packets, 800 observations, +100 decrypted messages and 800 ordinary/2,400 opted-in events with zero fixture +drops. These are bounded checks, not production capacity or losslessness claims. + +The final combined frontend passes native Pi build/lint and all **1,066 tests**. +Windows build/lint and 58 focused combined checks pass; the preceding route-only +tree also passed all 1,046 Windows and Pi tests. Public verification matches all +**23 assets**, both source archives and **26 boundaries**. Saved cards, expansion, +node inspection, French phone layout, LIVE delivery and pinned route pagination +were checked. The browser clipboard tool did not expose copied URL text; copy +construction is regression-tested and explicit shared-link navigation passes. +Physical Safari and a representative production-load soak remain unverified. + +## Preview recovery and follow-up + +The private full dump was restore-tested and checksum-verified on and off the Pi. +The phase-owned validation databases were retired after checking they were idle. +Only the Beacon app restarted; the other 23 containers were unchanged. Frontend +publication restarted none of the 24 containers. New traffic is preserved. + +The preview owner's guarded backend recovery is +`evidence/route-prefix-20260930/deploy-server.py rollback`, checkpoint +`routes-cutover-20260930T225443Z`. It restores server `61b0322a` / web `157525ef`, +rolling the experimental frontend back first if necessary. Frontend-only recovery +is `evidence/route-prefix-20260930/deploy-beacon-web.py rollback --evidence-dir route-prefix-20260930`, +checkpoint `web-20260930T232611Z`. This phase adds no migration or database restore. + +Daily compatibility checks run at 09:00 America/Toronto. They inspect upstream +changes and isolated trial merges, preserving active work and reporting meaningful +changes only. They do not push, rebase, deploy or ping reviewers automatically. +Use the [phased roadmap](post-140-roadmap.md) for subsequent work; node/trace +dashboards and deeper investigation are next, alongside the separate retained-window +decision. Atlas remains excluded from the stable 1.4.0 release even though it is +enabled in this experiment. diff --git a/app_documentation/post-140-roadmap.md b/app_documentation/post-140-roadmap.md index 7cc03ea..5d41ac8 100644 --- a/app_documentation/post-140-roadmap.md +++ b/app_documentation/post-140-roadmap.md @@ -12,12 +12,14 @@ or merge changes automatically. Version assignments remain with maintainers. The from this development queue; My Atlas remains excluded from that release, but is explicitly included in the experimental branch and Pi preview at the contributor's request. +[Current experimental build, validation and recovery](n30nex-test-preview.md). + ## Delivery order | Phase | Deliverable | Completion evidence | Status | |---|---|---|---| -| 1 — correctness suitable for 1.4.x | Saved-route hash-width consistency, then retention-aware time controls and remaining French/mobile/accessibility fixes | Route identity survives representation changes; evidence pagination and shared windows do not silently change path; raw and summary periods match available data | Implemented in experimental server PR #192 and the web branch; final Pi checks in progress | -| 2 — first feature after 1.4.0 | My Atlas saved-node monitoring | Carry the feature from web PR #97 into the experiment; saved identities/order survive; compact cards, expandable Heard by/statistics and existing entity links work in English/French on desktop/phone | Integrated into n30nex-test from the two feature commits; combined validation in progress | +| 1 — correctness suitable for 1.4.x | Saved-route hash-width consistency, then retention-aware time controls and remaining French/mobile/accessibility fixes | Route identity survives representation changes; evidence pagination and shared windows do not silently change path; raw and summary periods match available data | Delivered to the experimental Pi preview; server PR #192 stays draft | +| 2 — first feature after 1.4.0 | My Atlas saved-node monitoring | Carry the feature from web PR #97 into the experiment; saved identities/order survive; compact cards, expandable Heard by/statistics and existing entity links work in English/French on desktop/phone | Delivered in n30nex-test; all 1,066 native frontend tests and public checks pass | | 3 — node and trace investigation | Node dashboard, activity/type/signal/hop analysis, trace reception timeline and complete return navigation | Separate attributed node traffic from possible prefix matches; packet → route → node/observer → map links preserve selection, filters and Back | Queued | | 4 — find and compare | Bounded global entity search, saved views/filters, Atlas-node filters, channel activity and hearing context | Search/paging/share links agree; channel key/history availability is explicit; comparisons use aligned windows | Queued | | 5 — network structure | Observed route segments/alternatives, topology, distance, hash ambiguity and prefix/path inspection | Count evidence at the correct grain; separate observed ambiguity from static conflicts; use valid coordinates and show unresolved hops | Queued | From 762802919b957abba2db318f8a898281492fd053 Mon Sep 17 00:00:00 2001 From: n30nex Date: Wed, 30 Sep 2026 19:53:00 -0400 Subject: [PATCH 47/69] docs: record accepted upstream refresh and its recovery checkpoint --- app_documentation/n30nex-test-preview.md | 31 +++++++++++++++--------- 1 file changed, 20 insertions(+), 11 deletions(-) diff --git a/app_documentation/n30nex-test-preview.md b/app_documentation/n30nex-test-preview.md index e554aab..f49998c 100644 --- a/app_documentation/n30nex-test-preview.md +++ b/app_documentation/n30nex-test-preview.md @@ -11,7 +11,7 @@ has no new pull request. The stable 1.4.0 release handoff remains separate. | Repository | Experimental branch | Validated/deployed source | |---|---|---| -| Server | `n30nex-test`, `9505ad444268c552e08e46613ed8f2e0f8fd78a8` | Same revision; includes upstream dev `5b177d6f`, release parent #189 and the route-prefix fix in draft #192 | +| Server | `n30nex-test`, `af2604a610ee9758d448ffb1caa44b082182248f` | Same revision; includes accepted upstream dev `da6de40b` after #189 and the route-prefix fix in draft #192 | | Web | `n30nex-test`, `dc3268f7468bd9b3ae280243411c231fde6dd6ae` | Built as `b99b6e779d7bc4ec47e0e5202191d6aac9510ee0`; both have exactly the same `29d2513fe6f57f2aada079977e8eed40d72bc1dd` source tree | | Docs | `n30nex-test`, draft #7 | Active phased roadmap, experiment contract and this record | @@ -26,7 +26,8 @@ has no new pull request. The stable 1.4.0 release handoff remains separate. beside Observers. Saved full-key cards, activity bars, signal meters, expandable Heard by and statistics work in English and French. The cards remain bounded samples of up to 200 origin-key reports, not complete node totals. -- The ingestion status-text and logging corrections from upstream `5b177d6f` are +- The ingestion status-text and logging corrections, and the maintainer publishing + policy from upstream `da6de40b`, are included. Public admin/backup, automatic zones and foreign classification remain disabled; saved configuration and retention are unchanged. @@ -43,10 +44,12 @@ keeps its actual b99b6e7 identity and matching source archive; it is not relabel Native Go/PostgreSQL tests and server CI pass. The route regression changes widths between pages and checks shared links, repeats, malformed selectors, missing and colliding identities, and exact indexed plans. The 200,000-row fixture used the -existing index under custom and generic plans (0.097/0.090 ms execution in this -fixture). The 3,200-input replay retained the expected 100 packets, 800 observations, +existing index under both custom and generic plans. The 3,200-input replay retained the expected 100 packets, 800 observations, 100 decrypted messages and 800 ordinary/2,400 opted-in events with zero fixture -drops. These are bounded checks, not production capacity or losslessness claims. +drops. The post-upstream-refresh replay first missed its 20-second drain deadline +under the resource limit; the unchanged serialized retry passed in 14.40 seconds. +Both receipts are retained. These are bounded checks, not production capacity or +losslessness claims. The final combined frontend passes native Pi build/lint and all **1,066 tests**. Windows build/lint and 58 focused combined checks pass; the preceding route-only @@ -64,12 +67,18 @@ The phase-owned validation databases were retired after checking they were idle. Only the Beacon app restarted; the other 23 containers were unchanged. Frontend publication restarted none of the 24 containers. New traffic is preserved. -The preview owner's guarded backend recovery is -`evidence/route-prefix-20260930/deploy-server.py rollback`, checkpoint -`routes-cutover-20260930T225443Z`. It restores server `61b0322a` / web `157525ef`, -rolling the experimental frontend back first if necessary. Frontend-only recovery -is `evidence/route-prefix-20260930/deploy-beacon-web.py rollback --evidence-dir route-prefix-20260930`, -checkpoint `web-20260930T232611Z`. This phase adds no migration or database restore. +The latest backend recovery is +`evidence/experimental-refresh-20260930/deploy-server.py rollback`, checkpoint +`experimental-refresh-20260930T234341Z`. It restores server `9505ad44` while keeping +Atlas frontend `b99b6e77` and new traffic. To return all the way to the recorded +1.4.0 preview, run that rollback first, then +`evidence/route-prefix-20260930/deploy-server.py rollback`, which restores server +`61b0322a` / web `157525ef` and uses checkpoint `routes-cutover-20260930T225443Z`. +The feature phase's frontend-only recovery is +`evidence/route-prefix-20260930/deploy-beacon-web.py rollback --evidence-dir route-prefix-20260930`, +checkpoint `web-20260930T232611Z`; it is paired with server 9505ad44, so use it after +the latest backend rollback. Neither phase adds a schema change or restores an old +database over new traffic. Daily compatibility checks run at 09:00 America/Toronto. They inspect upstream changes and isolated trial merges, preserving active work and reporting meaningful From 0e83501cc212c71c081dcf661e067fa0fdb8f081 Mon Sep 17 00:00:00 2001 From: n30nex Date: Wed, 30 Sep 2026 20:18:52 -0400 Subject: [PATCH 48/69] docs: record proposed 2.0 baseline and synchronized version policy --- ROADMAP.md | 16 ++++++-- app_documentation/n30nex-test-preview.md | 13 ++++--- app_documentation/post-140-roadmap.md | 48 ++++++++++++++++++++---- 3 files changed, 60 insertions(+), 17 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index 6803994..d27b605 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,14 +1,22 @@ # Beacon parity and analytics roadmap -## Active post-1.4 phased work — 30 September +## Active experimental work and proposed 2.0 baseline — 30 September + +The latest maintainer discussion ends at **2.0.0**, with synchronized server/web +major/minor versions and independent patch levels. Flattened migrations and a clean +start are proposed but have not landed in the checked upstream refs. The next step +is to validate that published baseline on a separate database before extending the +feature work. The discussion is not authorization to wipe the Pi or reuse an old +migration journal with a new baseline. See the [version and migration direction](app_documentation/post-140-roadmap.md#maintainer-direction--30-september-2026). The contributor approved proceeding toward and beyond CoreScope feature parity. -The sequence is **saved-route correctness (#183) → My Atlas (#97, experimental now; release after 1.4.0) +The sequence is **validate the upcoming 2.0 baseline → preserve saved-route correctness (#183) and experimental My Atlas (#97) → node/trace dashboards and investigation → search/saved views/channel analytics → topology/distance/hash tools → bounded replay/reach/timing → regional crossing evidence**. -Phases 1 and 2 are combined on the requested `n30nex-test` branches in server, web and docs. The Pi preview runs this experimental composition, with daily upstream dev/main compatibility checks and no review pings. The route fix and Atlas are validated; node/trace dashboards and deeper investigation are next. Each phase ends in focused reviewable PRs, exact Pi validation, +Phases 1 and 2 are combined on the requested `n30nex-test` branches in server, web and docs. The Pi preview runs this experimental composition, with daily upstream dev/main compatibility checks and no review pings. The route fix and Atlas are validated; node/trace dashboards follow baseline validation. Each phase ends in focused reviewable PRs, exact Pi validation, updated source/changelog and retained rollback; release numbers remain a maintainer decision. -The current 1.4.0 handoff remains recorded below, with Atlas excluded. +The earlier 1.4.0 handoff remains recorded below as historical evidence. Atlas stays +experimental; its inclusion in a stable 2.0 release is not yet decided. [Phases, contracts and acceptance](app_documentation/post-140-roadmap.md) · [Current experimental preview](app_documentation/n30nex-test-preview.md). diff --git a/app_documentation/n30nex-test-preview.md b/app_documentation/n30nex-test-preview.md index f49998c..461a4d1 100644 --- a/app_documentation/n30nex-test-preview.md +++ b/app_documentation/n30nex-test-preview.md @@ -4,7 +4,10 @@ Verified 30 September 2026. The contributor requested separate `n30nex-test` branches, Atlas in the Pi preview, and daily checks against upstream dev/main. This work is experimental. Existing server #192 and docs #7 remain drafts; do not request reviews, mention reviewers or promote the work until asked. The web branch -has no new pull request. The stable 1.4.0 release handoff remains separate. +has no new pull request. The earlier 1.4.0 handoff remains a separate historical +checkpoint. The maintainer's proposed 2.0 baseline and synchronized version policy +are recorded in the [roadmap](post-140-roadmap.md); they have not changed this +running build or authorized a database reset. [Open My Atlas](https://canadaverse.org/beacon-dev/?tab=MyAtlas) · [Corresponding source and changelog](https://canadaverse.org/beacon-dev/source.html) @@ -83,7 +86,7 @@ database over new traffic. Daily compatibility checks run at 09:00 America/Toronto. They inspect upstream changes and isolated trial merges, preserving active work and reporting meaningful changes only. They do not push, rebase, deploy or ping reviewers automatically. -Use the [phased roadmap](post-140-roadmap.md) for subsequent work; node/trace -dashboards and deeper investigation are next, alongside the separate retained-window -decision. Atlas remains excluded from the stable 1.4.0 release even though it is -enabled in this experiment. +Use the [phased roadmap](post-140-roadmap.md) for subsequent work. Validate the +proposed upstream 2.0 baseline separately when it lands, preserving this database +and rollback; node/trace dashboards and deeper investigation follow. Atlas is +enabled in this experiment, with stable-release inclusion still undecided. diff --git a/app_documentation/post-140-roadmap.md b/app_documentation/post-140-roadmap.md index 5d41ac8..15ab1fd 100644 --- a/app_documentation/post-140-roadmap.md +++ b/app_documentation/post-140-roadmap.md @@ -1,16 +1,47 @@ -# Beacon after 1.4: parity and further development +# Beacon experimental roadmap: 2.0 baseline and feature parity + +## Maintainer direction — 30 September 2026 + +The contributor shared a maintainer discussion whose final proposed target is +**2.0.0**, superseding the earlier 1.7.0 suggestion and 1.4.0 planning target. +This is planning evidence, not a published release or permission to reset data. +At this check, upstream server dev remains `da6de40b`, web dev `e2d272e0`, and +docs main `e671ee45`; the announced migration reset has not landed in those refs. +The existing 1.4 records below and elsewhere remain dated validation checkpoints. + +The proposed version policy is shared **major/minor** versions for server and web, +with independent patch levels. Medium/large features advance the minor version, +fixes advance the patch version, and major changes or a deliberate new baseline +advance the major version. There is no automatic major bump after a fixed number +of minor releases. Maintainers choose release versions and tags. + +The maintainer plans to flatten historical migrations and require a clean start. +Once that work lands, inspect its actual startup/upgrade contract and validate the +new baseline on a separate disposable database. Preserve the current Pi database, +configuration, source and rollback. Do not apply a flattened migration history to +the existing database or reset it automatically. A fresh preview/production switch +needs a separate cutover decision after the candidate is concrete and tested. +Any retained history/import requirements must be decided explicitly; a new empty +database does not contain the old 30-day summaries. + +The next step is baseline compatibility and recovery validation, then the queued +node/trace work. Atlas stays enabled in the experiment; its inclusion in the stable +2.0 release has not been decided. The existing daily job checks and reports only; +it does not automatically rebase, push, deploy or reset databases. + +## Experimental delivery scope Approved direction, 30 September 2026: deliver useful CoreScope feature parity in small validated phases, then extend Beacon's regional and evidence-based analysis. -Post-1.4 work lives on the explicitly requested `n30nex-test` branches in server, +Development work lives on the explicitly requested `n30nex-test` branches in server, web and docs, with focused pull requests and the Pi preview as validation. The branch is experimental: keep existing PRs in draft and do not request or ping for review until the contributor asks. Daily upstream dev/main compatibility checks are scheduled; inspect dirty work first, use isolated trial merges, and report only new changes, conflicts or required decisions. Routine checks do not push, deploy -or merge changes automatically. Version assignments remain with maintainers. The 1.4.0 release handoff is separate -from this development queue; My Atlas remains excluded from that release, but is explicitly included in the -experimental branch and Pi preview at the contributor's request. +or merge changes automatically. Version assignments remain with maintainers. The +earlier 1.4.0 handoff is historical and separate from this development queue. +My Atlas is explicitly included in the experimental branch and Pi preview. [Current experimental build, validation and recovery](n30nex-test-preview.md). @@ -18,9 +49,10 @@ experimental branch and Pi preview at the contributor's request. | Phase | Deliverable | Completion evidence | Status | |---|---|---|---| -| 1 — correctness suitable for 1.4.x | Saved-route hash-width consistency, then retention-aware time controls and remaining French/mobile/accessibility fixes | Route identity survives representation changes; evidence pagination and shared windows do not silently change path; raw and summary periods match available data | Delivered to the experimental Pi preview; server PR #192 stays draft | -| 2 — first feature after 1.4.0 | My Atlas saved-node monitoring | Carry the feature from web PR #97 into the experiment; saved identities/order survive; compact cards, expandable Heard by/statistics and existing entity links work in English/French on desktop/phone | Delivered in n30nex-test; all 1,066 native frontend tests and public checks pass | -| 3 — node and trace investigation | Node dashboard, activity/type/signal/hop analysis, trace reception timeline and complete return navigation | Separate attributed node traffic from possible prefix matches; packet → route → node/observer → map links preserve selection, filters and Back | Queued | +| Next — upstream 2.0 baseline | Inspect synchronized versioning and the flattened migration contract when published | Validate an isolated fresh database, API/feature compatibility, available history and recoverable cutover; preserve the running Pi database | Awaiting upstream implementation | +| 1 — correctness | Saved-route hash-width consistency, then retention-aware time controls and remaining French/mobile/accessibility fixes | Route identity survives representation changes; evidence pagination and shared windows do not silently change path; raw and summary periods match available data | Route fix delivered to the experimental Pi preview; remaining controls/polish are separate; server PR #192 stays draft | +| 2 — experimental Atlas | My Atlas saved-node monitoring | Carry the feature from web PR #97 into the experiment; saved identities/order survive; compact cards, expandable Heard by/statistics and existing entity links work in English/French on desktop/phone | Delivered in n30nex-test; stable-release inclusion remains undecided | +| 3 — node and trace investigation | Node dashboard, activity/type/signal/hop analysis, trace reception timeline and complete return navigation | Separate attributed node traffic from possible prefix matches; packet → route → node/observer → map links preserve selection, filters and Back | Queued after baseline validation | | 4 — find and compare | Bounded global entity search, saved views/filters, Atlas-node filters, channel activity and hearing context | Search/paging/share links agree; channel key/history availability is explicit; comparisons use aligned windows | Queued | | 5 — network structure | Observed route segments/alternatives, topology, distance, hash ambiguity and prefix/path inspection | Count evidence at the correct grain; separate observed ambiguity from static conflicts; use valid coordinates and show unresolved hops | Queued | | 6 — history and reach | Bounded retained map replay, observer reach and timing analysis, comparable fleet telemetry | Replay preserves ordering and retention limits; confirmed identities are separate from unresolved prefixes; timing/counter gaps are not labelled packet loss | Queued | From 6b9ce19961f64379b8a623b3df5b03018e17211d Mon Sep 17 00:00:00 2001 From: n30nex Date: Wed, 30 Sep 2026 20:22:31 -0400 Subject: [PATCH 49/69] docs: record proposed dev and production retention policies --- ROADMAP.md | 5 +++++ app_documentation/n30nex-test-preview.md | 3 +++ app_documentation/post-140-roadmap.md | 17 +++++++++++++++++ 3 files changed, 25 insertions(+) diff --git a/ROADMAP.md b/ROADMAP.md index d27b605..e267b1c 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -9,6 +9,11 @@ is to validate that published baseline on a separate database before extending t feature work. The discussion is not authorization to wipe the Pi or reuse an old migration journal with a new baseline. See the [version and migration direction](app_documentation/post-140-roadmap.md#maintainer-direction--30-september-2026). +A follow-up proposes 24-hour raw retention on dev and 7 days in production, and +reports that dev has moved to rotational storage. No retention change is applied +to the Pi by this discussion. Chart windows need to follow the effective retention +of their deployment; summary-backed history remains distinct from raw records. + The contributor approved proceeding toward and beyond CoreScope feature parity. The sequence is **validate the upcoming 2.0 baseline → preserve saved-route correctness (#183) and experimental My Atlas (#97) → node/trace dashboards and investigation → search/saved views/channel analytics diff --git a/app_documentation/n30nex-test-preview.md b/app_documentation/n30nex-test-preview.md index 461a4d1..e408317 100644 --- a/app_documentation/n30nex-test-preview.md +++ b/app_documentation/n30nex-test-preview.md @@ -8,6 +8,9 @@ has no new pull request. The earlier 1.4.0 handoff remains a separate historical checkpoint. The maintainer's proposed 2.0 baseline and synchronized version policy are recorded in the [roadmap](post-140-roadmap.md); they have not changed this running build or authorized a database reset. +The later 24-hour dev / 7-day production raw-retention proposal is also not applied +to this Pi preview; its recorded policy remains 72-hour raw packets and 30-day +summaries pending an explicit configuration decision. [Open My Atlas](https://canadaverse.org/beacon-dev/?tab=MyAtlas) · [Corresponding source and changelog](https://canadaverse.org/beacon-dev/source.html) diff --git a/app_documentation/post-140-roadmap.md b/app_documentation/post-140-roadmap.md index 15ab1fd..4f78d96 100644 --- a/app_documentation/post-140-roadmap.md +++ b/app_documentation/post-140-roadmap.md @@ -29,6 +29,23 @@ node/trace work. Atlas stays enabled in the experiment; its inclusion in the sta 2.0 release has not been decided. The existing daily job checks and reports only; it does not automatically rebase, push, deploy or reset databases. +A follow-up maintainer discussion proposes **24-hour raw packets on dev** and +**7-day raw packets in production**, to keep development data and migrations +smaller. These are environment-specific proposals, not applied settings. They do +not authorize shortening the Pi preview's existing 72-hour raw retention or +purging its data. The previously agreed 30-day summary policy has not been changed +by this discussion. Validate the eventual configuration per deployment; offer raw +history windows that match that retention, with longer windows only where durable +summaries support them. The old fixed 24h/3d decision must be reconciled with this +proposed deployment split rather than applied globally. + +The maintainer also reports that dev now uses rotational storage. Treat that as +reported environment context, not a measured cause of any latency change; compare +performance with storage, data volume and workload identified. Keep the source/ +changelog as the release audit trail. When a new baseline is accepted, distinguish +its included work from later experimental changes while retaining older history +and matching source downloads. + ## Experimental delivery scope Approved direction, 30 September 2026: deliver useful CoreScope feature parity in From 4014ed29d7a4ced946d3325779650e6be86d08b0 Mon Sep 17 00:00:00 2001 From: n30nex Date: Wed, 30 Sep 2026 22:34:41 -0400 Subject: [PATCH 50/69] docs: record Mesh Pulse preview and prepared 2.0 integration --- app_documentation/meshmapper-scope-import.md | 17 ++ app_documentation/n30nex-test-preview.md | 207 ++++++++++--------- app_documentation/post-140-roadmap.md | 66 +++++- 3 files changed, 193 insertions(+), 97 deletions(-) diff --git a/app_documentation/meshmapper-scope-import.md b/app_documentation/meshmapper-scope-import.md index 3d16ef8..4614662 100644 --- a/app_documentation/meshmapper-scope-import.md +++ b/app_documentation/meshmapper-scope-import.md @@ -24,3 +24,20 @@ Operator logs under `component=meshmapper.scopes` show the IATA, source, active Catalogue counts remain source metadata. They never create Beacon packet counts, observer associations or node defaults. Actual packet-code matching supplies those associations through the existing ingest path. New names affect subsequent packets; no historical scan is started. Channel message tags and import-status UI are separate follow-ups. An invalid saved catalogue is logged and excluded until a valid refresh; other sources and manual keys still load. Database access failures remain startup errors. Disabling the importer restores manual-only matching. Previously imported identities remain visible in `/scopes` and historical records; visibility does not imply an active matcher candidate. Validate and back up PostgreSQL before applying the migration. Rollback to an older server must restore its matching pre-import database and configuration as well as its binary; an older matcher does not understand imported-only ownership or regional candidate limits. + +## Experimental catalogue view + +`GET /api/v1/scope-catalogues` returns only currently configured importer sources, +with normalized case-sensitive names, regional counts, monitored/wardriving flags, +source URL and generation/check/freshness times. No HTTP fetch or database query is +triggered by this endpoint. Disabled imports return an empty list; failed refreshes +retain the previous valid metadata with `lastError`. A past `freshUntil`, an error +or `checkedAt=0` must be shown as stale/unavailable. This response never assigns a +scope to individual nodes or links. Mesh Pulse separately shows Beacon's advertised +node defaults and packet-matched scopes. Older servers may return 404; the topology +continues without catalogue enrichment. + +The experimental node-list response now preserves `defaultScope`, which the list +query already selected. This is packet-derived node metadata, distinct from the +regional catalogue and its repeater counts. No per-node request or history scan is +needed for topology scope context. diff --git a/app_documentation/n30nex-test-preview.md b/app_documentation/n30nex-test-preview.md index e408317..27c433a 100644 --- a/app_documentation/n30nex-test-preview.md +++ b/app_documentation/n30nex-test-preview.md @@ -1,95 +1,116 @@ # Experimental n30nex-test preview -Verified 30 September 2026. The contributor requested separate `n30nex-test` -branches, Atlas in the Pi preview, and daily checks against upstream dev/main. -This work is experimental. Existing server #192 and docs #7 remain drafts; do not -request reviews, mention reviewers or promote the work until asked. The web branch -has no new pull request. The earlier 1.4.0 handoff remains a separate historical -checkpoint. The maintainer's proposed 2.0 baseline and synchronized version policy -are recorded in the [roadmap](post-140-roadmap.md); they have not changed this -running build or authorized a database reset. -The later 24-hour dev / 7-day production raw-retention proposal is also not applied -to this Pi preview; its recorded policy remains 72-hour raw packets and 30-day -summaries pending an explicit configuration decision. - -[Open My Atlas](https://canadaverse.org/beacon-dev/?tab=MyAtlas) · -[Corresponding source and changelog](https://canadaverse.org/beacon-dev/source.html) - -| Repository | Experimental branch | Validated/deployed source | -|---|---|---| -| Server | `n30nex-test`, `af2604a610ee9758d448ffb1caa44b082182248f` | Same revision; includes accepted upstream dev `da6de40b` after #189 and the route-prefix fix in draft #192 | -| Web | `n30nex-test`, `dc3268f7468bd9b3ae280243411c231fde6dd6ae` | Built as `b99b6e779d7bc4ec47e0e5202191d6aac9510ee0`; both have exactly the same `29d2513fe6f57f2aada079977e8eed40d72bc1dd` source tree | -| Docs | `n30nex-test`, draft #7 | Active phased roadmap, experiment contract and this record | - -## Included work - -- Saved routes update the current processed prefix representation without changing - their stable IATA/node-chain identity. New cursors and copied links pin one exact - width/path and time window; unrelated bytes or missing identities cannot silently - select another chain. The existing indexed observation query and schema are reused. -- My Atlas includes the two feature commits from web #97, adapted to the current - upstream navigation and translations. Existing tab order stays intact, with Atlas - beside Observers. Saved full-key cards, activity bars, signal meters, expandable - Heard by and statistics work in English and French. The cards remain bounded - samples of up to 200 origin-key reports, not complete node totals. -- The ingestion status-text and logging corrections, and the maintainer publishing - policy from upstream `da6de40b`, are - included. Public admin/backup, automatic zones and foreign classification remain - disabled; saved configuration and retention are unchanged. - -## Compatibility and validation - -Trial merges are clean for server and web against upstream dev and main, and for -docs against main. Web main `5ac36ce` was squash-created from `6daf342c` in release -PR #27. Both have the identical `03ab89640d90d77bc844b59b4945c4ad899f9cf0` tree; -the original is already an ancestor of current dev. A source-preserving ancestry -merge records that fact and removes the otherwise inherited 45 merge conflicts. -No source file was replaced during that reconciliation. The deployed web build -keeps its actual b99b6e7 identity and matching source archive; it is not relabelled. - -Native Go/PostgreSQL tests and server CI pass. The route regression changes widths -between pages and checks shared links, repeats, malformed selectors, missing and -colliding identities, and exact indexed plans. The 200,000-row fixture used the -existing index under both custom and generic plans. The 3,200-input replay retained the expected 100 packets, 800 observations, -100 decrypted messages and 800 ordinary/2,400 opted-in events with zero fixture -drops. The post-upstream-refresh replay first missed its 20-second drain deadline -under the resource limit; the unchanged serialized retry passed in 14.40 seconds. -Both receipts are retained. These are bounded checks, not production capacity or -losslessness claims. - -The final combined frontend passes native Pi build/lint and all **1,066 tests**. -Windows build/lint and 58 focused combined checks pass; the preceding route-only -tree also passed all 1,046 Windows and Pi tests. Public verification matches all -**23 assets**, both source archives and **26 boundaries**. Saved cards, expansion, -node inspection, French phone layout, LIVE delivery and pinned route pagination -were checked. The browser clipboard tool did not expose copied URL text; copy -construction is regression-tested and explicit shared-link navigation passes. -Physical Safari and a representative production-load soak remain unverified. - -## Preview recovery and follow-up - -The private full dump was restore-tested and checksum-verified on and off the Pi. -The phase-owned validation databases were retired after checking they were idle. -Only the Beacon app restarted; the other 23 containers were unchanged. Frontend -publication restarted none of the 24 containers. New traffic is preserved. - -The latest backend recovery is -`evidence/experimental-refresh-20260930/deploy-server.py rollback`, checkpoint -`experimental-refresh-20260930T234341Z`. It restores server `9505ad44` while keeping -Atlas frontend `b99b6e77` and new traffic. To return all the way to the recorded -1.4.0 preview, run that rollback first, then -`evidence/route-prefix-20260930/deploy-server.py rollback`, which restores server -`61b0322a` / web `157525ef` and uses checkpoint `routes-cutover-20260930T225443Z`. -The feature phase's frontend-only recovery is -`evidence/route-prefix-20260930/deploy-beacon-web.py rollback --evidence-dir route-prefix-20260930`, -checkpoint `web-20260930T232611Z`; it is paired with server 9505ad44, so use it after -the latest backend rollback. Neither phase adds a schema change or restores an old -database over new traffic. - -Daily compatibility checks run at 09:00 America/Toronto. They inspect upstream -changes and isolated trial merges, preserving active work and reporting meaningful -changes only. They do not push, rebase, deploy or ping reviewers automatically. -Use the [phased roadmap](post-140-roadmap.md) for subsequent work. Validate the -proposed upstream 2.0 baseline separately when it lands, preserving this database -and rollback; node/trace dashboards and deeper investigation follow. Atlas is -enabled in this experiment, with stable-release inclusion still undecided. +The Pi preview now includes **Mesh Pulse**, a 3D topology with animated live packet +paths, alongside My Atlas and pinned route evidence. Work remains on `n30nex-test`. +Server #192 and docs #7 stay draft, with no new review requests. There is no new +web PR, stable release or production-host change. + +[Open Mesh Pulse](https://canadaverse.org/beacon-dev/?tab=Topology) · +[YOW and scope context](https://canadaverse.org/beacon-dev/?tab=Topology&topoRegion=YOW) · +[Source and changelog](https://canadaverse.org/beacon-dev/source.html) + +| Repository | Preview source | +|---|---| +| Server | `644960e4a6d6a1c02e10ace182f6d335358d3f44` — cached catalogue API and the node-list default-scope mapping correction | +| Web | `4b2497dfc6e8adba145c6c660daebfe0fec5e648` — Mesh Pulse, Atlas and route evidence | +| Docs | `n30nex-test`, draft #7 — roadmap, contracts and this record | + +## Mesh Pulse + +- Native canvas perspective projection, stable IATA groups, orbit/zoom, keyboard + controls and node-type colours. Unlocated nodes are included. The finder searches + loaded names and public keys; selection highlights known neighbours. +- Live packet reports animate only adjacent, single-candidate resolved hops. Missing + or ambiguous identities leave gaps. Packet, node and reporting-observer actions + open Beacon's existing investigation panels. +- Cross-IATA links and matching advertised default scopes have separate styles. + MeshMapper regional catalogue metadata annotates matching scope context, without + assigning aggregate counts to individual repeaters. The API lacks per-repeater + identities. YOW is the currently configured source; other regions retain Beacon's + observed defaults and explicit unavailable-catalogue state. +- The node-list correction exposes a field its SQL already selected. It adds no + query per node. `/scope-catalogues` serves the existing importer's immutable cache; + viewing or refreshing this page does not initiate upstream MeshMapper HTTP. +- Bounded to 2,000 nodes, 5,000 stored neighbour links, 2,000 reports per browser + 60-second window and 64 simultaneous animated paths. Drawing is capped at 30 fps, + UI updates are coalesced, and the tab is lazy-loaded with no new dependencies. + Pause, reduced motion, hidden-tab cleanup and English/French controls are included. + +Groups use the latest receiving IATA, not geographic position. Stored neighbour +links have no selected time window. Motion illustrates a path, not measured RF +travel time, delivery or loss. Reports count packet/observer pairs received in this +view, with windowed deduplication and explicit pauses/reconnect gaps. + +## Validation + +The deployed frontend passed native Pi build/lint and **1,073 tests in 122 files**. +Windows tests and focused lifecycle, ambiguity, capacity, cancellation and unknown +payload tests pass. Desktop and French phone layouts, keyboard orbit, reduced motion, +node/observer return, retained packet drill-down, pause/resume and live motion were +checked. At the full-mesh browser check, 1,958 loaded nodes and 4,015 links were shown. +These are a dated observation, not permanent totals. + +Native Go/PostgreSQL tests pass for the server, including scope cache freshness, +concurrent snapshot readers, disabled/invalid sources and node-list scope projection. +The first combined replay passed all 3,200 inputs in 12.73 seconds after frontend +validation was serialized. The earlier missing-fixture setup failure and concurrent +20-second drain timeout are retained in the evidence. The final response-mapping +patch passed the native suite and replay in 16.28 seconds, with all 3,200 inputs and zero fixture drops. These fixtures do +not establish universal losslessness or production capacity. + +Public verification checks all 24 frontend files, both corresponding sources, +26 boundaries, the Public channel and both MQTT feeds. The scope catalogue check +returned nine YOW names. Source generation/check times and stale errors remain +visible. The short initial runtime sample showed app CPU of 3.22–6.25% and about +40–42 MiB memory. Database activity varied, including a brief one-core spike; a +follow-up sample was lower. A 355-second full-mesh browser sample measured about +19.5% main-thread task time and 19.3 MB JS heap. These samples are not a controlled +before/after benchmark or a physical-phone performance certification. + +## Upstream 2.0 landed during this phase + +Upstream server `3ac4035` replaces the migration chain with `001_baseline.sql` and +explicitly refuses legacy databases. Web `6de673b` prepares 2.0.0 and adds the +maintainer's analyzer/scope corrections. The live preview retains its 1.x database +and source pair; no history reset or migration-ledger relabelling was performed. + +The new upstream bases conflict with the legacy preview branches. Clean integration +commits are prepared **locally and separately** on `codex/topology-2-integration`: +server `835d1cdd4887576a6ca0cb687ef8077b73640f33`, web +`f2317febb5775234c5b684115685871544aecdff`. +Trial merges pass against current dev and main. The server was checked against a +separate fresh PostgreSQL database, including baseline installation, refusal of the +old ledger, pinned route evidence and the scope changes. Web build and 47 affected +checks pass. These integration commits are not deployed or submitted for review. + +Server CI passed on the initial `a5c8d28` head. The later legacy head has no new +GitHub checks because the upstream migration change conflicts with its base; do +not describe that head as CI-green or merge-ready. The separately prepared +integration resolves those conflicts. Choosing a fresh preview database and the +availability of retained history is the next cutover decision. The raw-retention +proposal (24h dev / 7d production) has not changed this Pi's 72h raw / 30d summaries. + +## Recovery + +All backups remain private, restored and checksum-verified on and off the Pi. +The response-mapping fix uses +`evidence/topology-scope-list-20261001/deploy-server.py rollback`, checkpoint +`scope-list-20261001T020111Z`, to restore server `a5c8d28` with web `c5265037`. +To undo the whole Mesh Pulse phase, run that recovery first, then +`evidence/mesh-topology-20261001/deploy-server.py rollback`, checkpoint +`mesh-topology-20261001T010811Z`, restoring `af2604a6` / `b99b6e77`. + +The latest frontend-only recovery is +`evidence/topology-scope-list-20261001/deploy-beacon-web.py rollback --evidence-dir topology-scope-list-20261001`, +checkpoint `web-20261001T022604Z`, restoring web `c5265037` with server `644960e4`. +The backend rollback above applies that frontend recovery first when necessary. + +The earlier frontend-only rollback for Mesh Pulse is +`evidence/mesh-topology-20261001/deploy-beacon-web.py rollback --evidence-dir mesh-topology-20261001`, +checkpoint `web-20261001T014625Z`; it is paired with server a5c8d28, so undo the later +response-mapping fix first. All recoveries preserve new traffic. No schema or +configuration change was made, and unrelated containers were preserved. + +Daily checks at 09:00 America/Toronto remain checks only. See the +[phased roadmap](post-140-roadmap.md) for the 2.0 integration decision, constrained +live follow, label/detail refinement, shared display preferences and bounded replay. diff --git a/app_documentation/post-140-roadmap.md b/app_documentation/post-140-roadmap.md index 4f78d96..044c8ee 100644 --- a/app_documentation/post-140-roadmap.md +++ b/app_documentation/post-140-roadmap.md @@ -1,5 +1,16 @@ # Beacon experimental roadmap: 2.0 baseline and feature parity +## Current checkpoint — 1 October 2026 + +Mesh Pulse is deployed on the experimental Pi with Atlas and route evidence. The +node-list scope field is restored so stored defaults can enrich neighbour links. +Upstream server `3ac4035` and web `6de673b` landed during this phase. The 2.0 +integration is prepared separately and validated on a fresh test database; the live +preview keeps its legacy database and retained history. Promoting that integration +and choosing history availability is the next decision, before new baseline-dependent +features. The dated proposal below is retained as history and is superseded by +this implementation state. [Exact preview and recovery](n30nex-test-preview.md). + ## Maintainer direction — 30 September 2026 The contributor shared a maintainer discussion whose final proposed target is @@ -24,8 +35,8 @@ needs a separate cutover decision after the candidate is concrete and tested. Any retained history/import requirements must be decided explicitly; a new empty database does not contain the old 30-day summaries. -The next step is baseline compatibility and recovery validation, then the queued -node/trace work. Atlas stays enabled in the experiment; its inclusion in the stable +Baseline compatibility and recovery validation still gate database-dependent +node/trace work. The contributor has since prioritized the frontend topology phase below. Atlas stays enabled in the experiment; its inclusion in the stable 2.0 release has not been decided. The existing daily job checks and reports only; it does not automatically rebase, push, deploy or reset databases. @@ -62,16 +73,63 @@ My Atlas is explicitly included in the experimental branch and Pi preview. [Current experimental build, validation and recovery](n30nex-test-preview.md). +## CartoLite-inspired preview phase — 1 October 2026 + +The contributor moved a graphical, animated topology experiment ahead of the +node/trace dashboards. Work stays on `n30nex-test`, including Atlas, without review +pings. This phase reuses existing node, neighbour and live packet data; it does not +need the proposed 2.0 database reset. Baseline compatibility remains a separate gate. + +Reference audit: [CartoLite](https://github.com/n30nex/CartoLite/tree/f2b4bdff314094c22749819ddc6817d545aa0fa0), +particularly [Netgraph](https://github.com/n30nex/CartoLite/blob/f2b4bdff314094c22749819ddc6817d545aa0fa0/docs/netgraph.md), +`netgraph/layout.ts`, `quality.ts`, `follow.ts`, `nodeInspector.ts` and `routeFocus.ts`. +CartoLite currently uses layered 2D canvases. Beacon adds a real perspective camera +and 3D node positions with native canvas drawing, without a new rendering dependency. +The implementation is original; CartoLite is a design reference, not an embedded app. + +| Priority | Useful CartoLite idea | Beacon adaptation and evidence boundary | State | +|---|---|---|---| +| Now | Stable area packing and transient packet overlays | Mesh Pulse: deterministic IATA clusters, orbit/zoom, node shapes, packet-type colours; preserve unknown hops instead of drawing invented links | Delivered on the experimental Pi preview | +| Now | Finder and neighbour spotlight | Search loaded full identities, highlight existing neighbour links, inspect nodes/packets/reporting observers through Beacon panels | Included in this phase | +| Now | Compact controls and bounded drawing work | Lazy tab, 2,000 nodes/5,000 links, at most 64 animated paths, 30 fps ceiling, reduced motion, pause/background cleanup, keyboard controls, English/French | Included in this phase | +| Now | Regional context | Differentiate cross-IATA links and matching advertised default scopes; show cached MeshMapper catalogue counts/freshness separately | Included in this phase | +| Next | Route focus and constrained live follow | Follow a selected node/observer/region without camera jumps; retain explicit exit and keyboard return | Planned after first topology feedback | +| Next | Display preferences | Declutter labels at overview scale; save topology camera/detail preferences, clear reset, cache static ink separately from traffic, measured adaptive quality and a reusable motion setting | Planned; measure before increasing rendering complexity | +| Later | Retained traffic replay | Bounded server cursor/window, visible gaps and retention, one time controller shared by map/topology | Requires retained-evidence API contract | +| Later | Cross-view selection and discovery | Shared node/route focus between Atlas, topology, map and investigation; bounded global search | Align with phases 3–4 | +| Optional | Sound and exhibition views | Opt-in sound only after accessibility/performance feedback; no automatic audio or copied decorative Labs assets | Deferred | + +[MeshMapper Scopes API](https://wiki.meshmapper.net/scopes-api/) was rechecked. +It is public, uses a five-minute cache/ETag and a 60 requests/minute/IP limit. It +returns names and regional counts, **not per-repeater identities**. Keep Beacon's +existing configured-source importer, conditional requests and manual fallback. +A read-only `/scope-catalogues` endpoint exposes its immutable cached metadata; +opening a graph never initiates upstream HTTP or packet-table scans. A missing, +failed, disabled or stale catalogue must not create or remove neighbour evidence. +Case-sensitive scope names and monitored zero-count entries remain intact. + +The [Zones API](https://wiki.meshmapper.net/zones-api/) lists published regional +URLs by country and serves nullable GeoJSON boundaries, cached hourly with ETags. +Use that discovery contract for a future configured-source expansion and map link; +keep geometric boundary crossings separate from receiving-IATA changes. + +Shared advertised defaults are context, not proof of forwarding. An IATA group is +based on the node's latest hearing region, not geographic coordinates. Packet +colours follow high-confidence adjacent resolved hops; animation speed is illustrative. +The existing Pi importer is configured for YOW. Other IATAs must say their catalogue +is unavailable until a published source is deliberately configured; do not invent +URLs or assign a group catalogue's counts to every member region. + ## Delivery order | Phase | Deliverable | Completion evidence | Status | |---|---|---|---| -| Next — upstream 2.0 baseline | Inspect synchronized versioning and the flattened migration contract when published | Validate an isolated fresh database, API/feature compatibility, available history and recoverable cutover; preserve the running Pi database | Awaiting upstream implementation | +| Separate gate — upstream 2.0 baseline | Inspect synchronized versioning and the flattened migration contract when published | Validate an isolated fresh database, API/feature compatibility, available history and recoverable cutover; preserve the running Pi database | Landed during this phase; isolated integration and fresh-database checks pass, cutover/history decision pending | | 1 — correctness | Saved-route hash-width consistency, then retention-aware time controls and remaining French/mobile/accessibility fixes | Route identity survives representation changes; evidence pagination and shared windows do not silently change path; raw and summary periods match available data | Route fix delivered to the experimental Pi preview; remaining controls/polish are separate; server PR #192 stays draft | | 2 — experimental Atlas | My Atlas saved-node monitoring | Carry the feature from web PR #97 into the experiment; saved identities/order survive; compact cards, expandable Heard by/statistics and existing entity links work in English/French on desktop/phone | Delivered in n30nex-test; stable-release inclusion remains undecided | | 3 — node and trace investigation | Node dashboard, activity/type/signal/hop analysis, trace reception timeline and complete return navigation | Separate attributed node traffic from possible prefix matches; packet → route → node/observer → map links preserve selection, filters and Back | Queued after baseline validation | | 4 — find and compare | Bounded global entity search, saved views/filters, Atlas-node filters, channel activity and hearing context | Search/paging/share links agree; channel key/history availability is explicit; comparisons use aligned windows | Queued | -| 5 — network structure | Observed route segments/alternatives, topology, distance, hash ambiguity and prefix/path inspection | Count evidence at the correct grain; separate observed ambiguity from static conflicts; use valid coordinates and show unresolved hops | Queued | +| 5 — network structure | Observed route segments/alternatives, topology, distance, hash ambiguity and prefix/path inspection | Count evidence at the correct grain; separate observed ambiguity from static conflicts; use valid coordinates and show unresolved hops | 3D topology and scope context delivered; distance/hash inspection remains queued | | 6 — history and reach | Bounded retained map replay, observer reach and timing analysis, comparable fleet telemetry | Replay preserves ordering and retention limits; confirmed identities are separate from unresolved prefixes; timing/counter gaps are not labelled packet loss | Queued | | Beyond parity | Regional boundary/scope crossing investigation and links between observations, discovered scopes and route changes | Each relationship links to retained evidence; distinguish reported locations from inferred paths and scope names from geography | Queued after supporting phases | From d87cc1137940a638fef94acb9458c0f4d1f55837 Mon Sep 17 00:00:00 2001 From: n30nex Date: Thu, 1 Oct 2026 19:17:49 -0400 Subject: [PATCH 51/69] docs: record connected topology preview and camera validation --- app_documentation/n30nex-test-preview.md | 71 ++++++++++++++++++------ app_documentation/post-140-roadmap.md | 30 ++++++---- 2 files changed, 73 insertions(+), 28 deletions(-) diff --git a/app_documentation/n30nex-test-preview.md b/app_documentation/n30nex-test-preview.md index 27c433a..57c0b07 100644 --- a/app_documentation/n30nex-test-preview.md +++ b/app_documentation/n30nex-test-preview.md @@ -12,14 +12,16 @@ web PR, stable release or production-host change. | Repository | Preview source | |---|---| | Server | `644960e4a6d6a1c02e10ace182f6d335358d3f44` — cached catalogue API and the node-list default-scope mapping correction | -| Web | `4b2497dfc6e8adba145c6c660daebfe0fec5e648` — Mesh Pulse, Atlas and route evidence | +| Web | `117ee99a60e6f7da352bb27a442eaae7f056706d` — connected layout, regional camera, saved routes and live trails; Atlas retained | | Docs | `n30nex-test`, draft #7 — roadmap, contracts and this record | ## Mesh Pulse -- Native canvas perspective projection, stable IATA groups, orbit/zoom, keyboard - controls and node-type colours. Unlocated nodes are included. The finder searches - loaded names and public keys; selection highlights known neighbours. +- Native canvas perspective projection with connection-driven regional placement, + separate islands sized for their populations, and connected-node spacing. Region + labels and the camera selector focus a region; Isolate loads it alone. Pan/orbit, + pinch/scroll zoom, keyboard controls, top/3D views and full screen are included. + Focus and isolation survive shared URLs and Back/reload. - Live packet reports animate only adjacent, single-candidate resolved hops. Missing or ambiguous identities leave gaps. Packet, node and reporting-observer actions open Beacon's existing investigation panels. @@ -31,24 +33,42 @@ web PR, stable release or production-host change. - The node-list correction exposes a field its SQL already selected. It adds no query per node. `/scope-catalogues` serves the existing importer's immutable cache; viewing or refreshing this page does not initiate upstream MeshMapper HTTP. -- Bounded to 2,000 nodes, 5,000 stored neighbour links, 2,000 reports per browser - 60-second window and 64 simultaneous animated paths. Drawing is capped at 30 fps, - UI updates are coalesced, and the tab is lazy-loaded with no new dependencies. - Pause, reduced motion, hidden-tab cleanup and English/French controls are included. - -Groups use the latest receiving IATA, not geographic position. Stored neighbour +- All paths is the default: every loaded neighbour link plus adjacent resolved + saved-route segments. The route window is 15 minutes by default, with one-hour + and 24-hour choices. Live trails persist for 60 seconds. Region bundles and + selected-node views are optional; live animations continue in every display mode. +- Bounded to 20,000 nodes, 60,000 recent routes, 100,000 links/live segments, + 10,000 reports per browser 60-second window and 512 simultaneous animations. + A reached limit is visible. Static ink is cached separately from live motion, + animation targets 30 fps, UI updates are coalesced and queries are cancellable. + Pause, reduced motion, hidden-tab cleanup and English/French controls remain. + No rendering dependency or database migration was added. + +Groups use the latest receiving IATA, not geographic position; isolation includes +all loaded nodes heard in that IATA. Connection-driven placement reduces clutter +but does not guarantee zero projected crossings in a dense, fully visible graph. +Curve height is presentation only, not physical altitude. Stored neighbour links have no selected time window. Motion illustrates a path, not measured RF travel time, delivery or loss. Reports count packet/observer pairs received in this view, with windowed deduplication and explicit pauses/reconnect gaps. ## Validation -The deployed frontend passed native Pi build/lint and **1,073 tests in 122 files**. -Windows tests and focused lifecycle, ambiguity, capacity, cancellation and unknown -payload tests pass. Desktop and French phone layouts, keyboard orbit, reduced motion, -node/observer return, retained packet drill-down, pause/resume and live motion were -checked. At the full-mesh browser check, 1,958 loaded nodes and 4,015 links were shown. -These are a dated observation, not permanent totals. +The exact frontend passed native Pi build/lint and **1,078 tests in 122 files**. +The final native assets passed desktop and French 390px browser checks: all-path +visibility, 160 simultaneous report animations, no static redraws during a sampled +second of animation, full screen, regional camera, pan/zoom/fit, isolation, +Back/reload and reduced motion. The public site returned 1,991 nodes and 4,220 +connections in the default window, or 5,041 connections after the 24-hour query +completed, with no cap warning. A fresh browser loaded the default view in 2.31s; +loading the longer route history took about 11s. These are observations, not a +load-capacity guarantee. The live check received 112 real reports, including 84 +with resolved paths, and no page errors. + +Windows builds and browser checks passed. One local Node 22 validation process +exited with an access violation; the unfinished Windows full suite was stopped. +The exact candidate's complete suite passed using Node 24.15.0 on the Pi. +Physical iPhone/Safari and sustained production-load qualification are not claimed. Native Go/PostgreSQL tests pass for the server, including scope cache freshness, concurrent snapshot readers, disabled/invalid sources and node-list scope projection. @@ -67,6 +87,23 @@ follow-up sample was lower. A 355-second full-mesh browser sample measured about 19.5% main-thread task time and 19.3 MB JS heap. These samples are not a controlled before/after benchmark or a physical-phone performance certification. +## Current frontend recovery — 1 October + +The connected-layout update changes only static frontend files. All 24 published +files and both corresponding-source archives match; the source/changelog is current. +Every existing container identity and restart count was unchanged by deployment. +Server `644960e4`, schema 045, configuration and the 72h raw/30d summary policy remain. + +Latest rollback is `evidence/topology-camera-20261001/deploy-beacon-web.py rollback +--evidence-dir topology-camera-20261001`, checkpoint `web-20261001T231151Z`. It restores +web `4b2497df` while keeping the server and new traffic. Use this frontend rollback +before the older backend recovery recipes below. + +The new patch applies cleanly to the prepared 2.0 web checkout. The whole experimental +branch still has 14 dev merge conflicts inherited from the 2.0 transition; main +trial-merges cleanly. The earlier prepared-integration receipts below apply to their +recorded upstream revisions and do not establish compatibility with newer dev. + ## Upstream 2.0 landed during this phase Upstream server `3ac4035` replaces the migration chain with `001_baseline.sql` and @@ -100,7 +137,7 @@ To undo the whole Mesh Pulse phase, run that recovery first, then `evidence/mesh-topology-20261001/deploy-server.py rollback`, checkpoint `mesh-topology-20261001T010811Z`, restoring `af2604a6` / `b99b6e77`. -The latest frontend-only recovery is +The previous frontend-only recovery is `evidence/topology-scope-list-20261001/deploy-beacon-web.py rollback --evidence-dir topology-scope-list-20261001`, checkpoint `web-20261001T022604Z`, restoring web `c5265037` with server `644960e4`. The backend rollback above applies that frontend recovery first when necessary. diff --git a/app_documentation/post-140-roadmap.md b/app_documentation/post-140-roadmap.md index 044c8ee..dc4435c 100644 --- a/app_documentation/post-140-roadmap.md +++ b/app_documentation/post-140-roadmap.md @@ -2,14 +2,20 @@ ## Current checkpoint — 1 October 2026 -Mesh Pulse is deployed on the experimental Pi with Atlas and route evidence. The -node-list scope field is restored so stored defaults can enrich neighbour links. -Upstream server `3ac4035` and web `6de673b` landed during this phase. The 2.0 -integration is prepared separately and validated on a fresh test database; the live -preview keeps its legacy database and retained history. Promoting that integration -and choosing history availability is the next decision, before new baseline-dependent -features. The dated proposal below is retained as history and is superseded by -this implementation state. [Exact preview and recovery](n30nex-test-preview.md). +Mesh Pulse includes Atlas, route evidence and the connected-layout/camera patch +`117ee99`. Regions and nodes are placed using actual connections; larger groups get +more space. All loaded paths remain visible by default. Region focus, isolation, +pan/orbit/zoom, top view and full screen work alongside live packet animation. +The topology now includes resolved saved routes, with 15-minute (default), one-hour +and 24-hour windows, plus 60-second live trails. Static drawing is cached separately. + +The 2.0 baseline has landed upstream. The latest compatibility check found source +conflicts with dev, including the separately prepared 2.0 integration; its earlier +clean-merge receipt is no longer current. This focused frontend patch applies cleanly +to that prepared checkout and still trial-merges cleanly with main. Refreshing the +overall integration and choosing a fresh-database/history cutover remain separate. +The live experiment keeps its legacy database and retained history. +[Exact preview, validation and recovery](n30nex-test-preview.md). ## Maintainer direction — 30 September 2026 @@ -91,10 +97,12 @@ The implementation is original; CartoLite is a design reference, not an embedded |---|---|---|---| | Now | Stable area packing and transient packet overlays | Mesh Pulse: deterministic IATA clusters, orbit/zoom, node shapes, packet-type colours; preserve unknown hops instead of drawing invented links | Delivered on the experimental Pi preview | | Now | Finder and neighbour spotlight | Search loaded full identities, highlight existing neighbour links, inspect nodes/packets/reporting observers through Beacon panels | Included in this phase | -| Now | Compact controls and bounded drawing work | Lazy tab, 2,000 nodes/5,000 links, at most 64 animated paths, 30 fps ceiling, reduced motion, pause/background cleanup, keyboard controls, English/French | Included in this phase | +| Now | Compact controls and bounded drawing work | Lazy tab, 20,000 nodes, 60,000 recent routes, 100,000 links/live segments, 10,000 reports and 512 animations; static ink cache, 30 fps ceiling, reduced motion, pause/background cleanup, English/French | Included in connected-layout patch | | Now | Regional context | Differentiate cross-IATA links and matching advertised default scopes; show cached MeshMapper catalogue counts/freshness separately | Included in this phase | -| Next | Route focus and constrained live follow | Follow a selected node/observer/region without camera jumps; retain explicit exit and keyboard return | Planned after first topology feedback | -| Next | Display preferences | Declutter labels at overview scale; save topology camera/detail preferences, clear reset, cache static ink separately from traffic, measured adaptive quality and a reusable motion setting | Planned; measure before increasing rendering complexity | +| Now | Connected layout and regional camera | Pull linked regions together, separate islands, spread connected nodes, preserve all paths, focus/isolate regions and fit/pan/orbit/zoom; full screen and shared focus URLs | Included in connected-layout patch | +| Now | Observed route coverage | Load adjacent resolved route segments for a selected window; keep missing-hop gaps, distinguish these from neighbour records and live reports | Included in connected-layout patch | +| Next | Constrained live follow | Follow a selected node/observer/region without camera jumps; retain explicit exit and keyboard return | Planned | +| Next | Display preferences | Save camera/detail preferences and add measured adaptive quality; static ink caching and label collision handling are implemented | Planned; measure before adding rendering complexity | | Later | Retained traffic replay | Bounded server cursor/window, visible gaps and retention, one time controller shared by map/topology | Requires retained-evidence API contract | | Later | Cross-view selection and discovery | Shared node/route focus between Atlas, topology, map and investigation; bounded global search | Align with phases 3–4 | | Optional | Sound and exhibition views | Opt-in sound only after accessibility/performance feedback; no automatic audio or copied decorative Labs assets | Deferred | From 72da97cea3e2d2217d7c2f3f27a32f68d1b202b5 Mon Sep 17 00:00:00 2001 From: n30nex Date: Thu, 1 Oct 2026 20:08:52 -0400 Subject: [PATCH 52/69] docs: record simplified topology preview and Atlas navigation --- app_documentation/n30nex-test-preview.md | 72 ++++++++++++++++++------ app_documentation/post-140-roadmap.md | 9 +++ 2 files changed, 63 insertions(+), 18 deletions(-) diff --git a/app_documentation/n30nex-test-preview.md b/app_documentation/n30nex-test-preview.md index 57c0b07..e294f6a 100644 --- a/app_documentation/n30nex-test-preview.md +++ b/app_documentation/n30nex-test-preview.md @@ -1,6 +1,6 @@ # Experimental n30nex-test preview -The Pi preview now includes **Mesh Pulse**, a 3D topology with animated live packet +The Pi preview now includes **Topology**, a 3D topology with animated live packet paths, alongside My Atlas and pinned route evidence. Work remains on `n30nex-test`. Server #192 and docs #7 stay draft, with no new review requests. There is no new web PR, stable release or production-host change. @@ -12,9 +12,31 @@ web PR, stable release or production-host change. | Repository | Preview source | |---|---| | Server | `644960e4a6d6a1c02e10ace182f6d335358d3f44` — cached catalogue API and the node-list default-scope mapping correction | -| Web | `117ee99a60e6f7da352bb27a442eaae7f056706d` — connected layout, regional camera, saved routes and live trails; Atlas retained | +| Web | `462d49e8afb2fabf54f20832318a0f7ae0a0d77d` — My Atlas first, simpler Topology controls and camera orientation correction | | Docs | `n30nex-test`, draft #7 — roadmap, contracts and this record | +## Topology UI and navigation — 1 October + +My Atlas is the far-left desktop tab and the first mobile tab. Nodes remains in the +mobile More menu. Topology uses a compact status/count header and one region selector. +The main toolbar contains region, route window, pause and node search. Display holds +path modes, camera presets, drag mode, animation, legend, sharing and refresh. + +The inspector opens when a node is selected or searched. Phones reuse the existing +focus-trapped details sheet. Closing details returns to the graph; search from full +screen first exits full screen so its input is visible and focused. Live activity, +reporters and packet details are expandable and continue receiving data while closed. +Scope metadata is fetched only when the inspector is open. + +The camera now looks down from above the mesh. Its projection previously placed the +camera below the plane even for Top view. Raised points are now nearer the camera; +pan follows the corrected axes, and Fit centres projected bounds before scaling. +Pan is the default drag action, Shift-drag rotates, and Display offers Top/3D views. +The graph stays visible while a new route window loads, with an updating indicator. + +All-path visibility, live packet animation, full screen, region focus/isolation and +English/French controls remain. This is a frontend-only update on `n30nex-test`. + ## Mesh Pulse - Native canvas perspective projection with connection-driven regional placement, @@ -54,21 +76,20 @@ view, with windowed deduplication and explicit pauses/reconnect gaps. ## Validation -The exact frontend passed native Pi build/lint and **1,078 tests in 122 files**. -The final native assets passed desktop and French 390px browser checks: all-path -visibility, 160 simultaneous report animations, no static redraws during a sampled -second of animation, full screen, regional camera, pan/zoom/fit, isolation, -Back/reload and reduced motion. The public site returned 1,991 nodes and 4,220 -connections in the default window, or 5,041 connections after the 24-hour query -completed, with no cap warning. A fresh browser loaded the default view in 2.31s; -loading the longer route history took about 11s. These are observations, not a -load-capacity guarantee. The live check received 112 real reports, including 84 -with resolved paths, and no page errors. - -Windows builds and browser checks passed. One local Node 22 validation process -exited with an access violation; the unfinished Windows full suite was stopped. -The exact candidate's complete suite passed using Node 24.15.0 on the Pi. -Physical iPhone/Safari and sustained production-load qualification are not claimed. +The exact frontend passed native Pi build/lint and **1,079 tests in 122 files**. +Windows build, changed-file lint and all 30 focused navigation/topology tests passed. +The above-mesh regression checks depth, apparent scale and screen position of raised +nodes; framing and pan tests cover desktop and portrait dimensions. + +The final native browser fixture verified Atlas first on desktop/mobile, one region +control, inspector focus (including search from full screen), French phone menus, +expanded activity, pause/resume, retained chart data during window loading, region +focus/isolation, reduced motion and 160 simultaneous animations. Static ink did not +repaint during the sampled second of packet animation. Public checks verified Atlas +navigation, Back, default all-path visibility and mobile focus/menu bounds. The +public browser loaded the view in 2.68s and received 138 real reports, 99 with +resolved paths, with no page errors. These are bounded observations, not a sustained +load-capacity guarantee. Physical iPhone/Safari qualification is not claimed. Native Go/PostgreSQL tests pass for the server, including scope cache freshness, concurrent snapshot readers, disabled/invalid sources and node-list scope projection. @@ -89,12 +110,27 @@ before/after benchmark or a physical-phone performance certification. ## Current frontend recovery — 1 October +The UI update changes static frontend files only. All 24 assets, both source archives +and the source/changelog match. All 24 existing containers were unchanged. Server +`644960e4`, schema 045, configuration and 72h raw/30d summary retention remain. + +Latest rollback is `evidence/topology-ux-20261001/deploy-beacon-web.py rollback +--evidence-dir topology-ux-20261001`, checkpoint `web-20261002T000512Z`. It restores +web `117ee99a` without changing the server or new traffic. Use it before the previous +connected-layout and backend recovery recipes below. The checkpoint uses UTC. + +The combined camera/UI patch applies cleanly to the prepared 2.0 web checkout. +The whole branch still has 14 dev conflicts at `f9ffb564`; main trial-merges cleanly. +The experiment remains on `n30nex-test`, with draft reviews held and no new web PR. + +## Previous connected-layout recovery — 1 October + The connected-layout update changes only static frontend files. All 24 published files and both corresponding-source archives match; the source/changelog is current. Every existing container identity and restart count was unchanged by deployment. Server `644960e4`, schema 045, configuration and the 72h raw/30d summary policy remain. -Latest rollback is `evidence/topology-camera-20261001/deploy-beacon-web.py rollback +That phase’s rollback is `evidence/topology-camera-20261001/deploy-beacon-web.py rollback --evidence-dir topology-camera-20261001`, checkpoint `web-20261001T231151Z`. It restores web `4b2497df` while keeping the server and new traffic. Use this frontend rollback before the older backend recovery recipes below. diff --git a/app_documentation/post-140-roadmap.md b/app_documentation/post-140-roadmap.md index dc4435c..a5bbfbe 100644 --- a/app_documentation/post-140-roadmap.md +++ b/app_documentation/post-140-roadmap.md @@ -2,6 +2,14 @@ ## Current checkpoint — 1 October 2026 +The deployed UI pass is web `462d49e`: My Atlas leads desktop and mobile navigation. +Topology has one region selector and a compact toolbar, an inspector that opens on +selection/search, a mobile details sheet and expandable Display/Live activity panels. +The camera now starts above the mesh; Fit centres the projected bounds and drag +defaults to pan, with Shift-drag rotating. All paths and live traffic remain available. +Route-window loading keeps the existing graph visible while the new data arrives. +There are no backend, database, configuration or retention changes. + Mesh Pulse includes Atlas, route evidence and the connected-layout/camera patch `117ee99`. Regions and nodes are placed using actual connections; larger groups get more space. All loaded paths remain visible by default. Region focus, isolation, @@ -100,6 +108,7 @@ The implementation is original; CartoLite is a design reference, not an embedded | Now | Compact controls and bounded drawing work | Lazy tab, 20,000 nodes, 60,000 recent routes, 100,000 links/live segments, 10,000 reports and 512 animations; static ink cache, 30 fps ceiling, reduced motion, pause/background cleanup, English/French | Included in connected-layout patch | | Now | Regional context | Differentiate cross-IATA links and matching advertised default scopes; show cached MeshMapper catalogue counts/freshness separately | Included in this phase | | Now | Connected layout and regional camera | Pull linked regions together, separate islands, spread connected nodes, preserve all paths, focus/isolate regions and fit/pan/orbit/zoom; full screen and shared focus URLs | Included in connected-layout patch | +| Now | Simpler navigation and controls | My Atlas first, one region picker, compact summary, on-demand inspector/activity, accessible mobile sheet, above-mesh camera and centred Fit | Included in UI pass `462d49e` | | Now | Observed route coverage | Load adjacent resolved route segments for a selected window; keep missing-hop gaps, distinguish these from neighbour records and live reports | Included in connected-layout patch | | Next | Constrained live follow | Follow a selected node/observer/region without camera jumps; retain explicit exit and keyboard return | Planned | | Next | Display preferences | Save camera/detail preferences and add measured adaptive quality; static ink caching and label collision handling are implemented | Planned; measure before adding rendering complexity | From 71125bcd51aa19b9ead8c49b530525f7979db59d Mon Sep 17 00:00:00 2001 From: n30nex Date: Fri, 2 Oct 2026 09:11:33 -0400 Subject: [PATCH 53/69] docs: prepare focused Atlas and Topology integration after 2.0 --- app_documentation/n30nex-test-preview.md | 9 +++ app_documentation/post-140-roadmap.md | 48 ++++++------ app_documentation/post-20-integration.md | 94 ++++++++++++++++++++++++ 3 files changed, 125 insertions(+), 26 deletions(-) create mode 100644 app_documentation/post-20-integration.md diff --git a/app_documentation/n30nex-test-preview.md b/app_documentation/n30nex-test-preview.md index e294f6a..de666ad 100644 --- a/app_documentation/n30nex-test-preview.md +++ b/app_documentation/n30nex-test-preview.md @@ -1,5 +1,14 @@ # Experimental n30nex-test preview +## Prepared branch versus deployed preview — 2 October + +The prepared 2.0 branch heads are server `ae328cb6` and web `9d3f9585`, including +upstream hourly rollups. They have not been deployed. The table below continues +to identify the actual running binaries/assets and their matching source offer. +[Post-2.0 feature packages and validation](post-20-integration.md) describes the +clean merge sequence. Daily compatibility checks remain paused; no database reset +or review request is authorized by this preparation. + The Pi preview now includes **Topology**, a 3D topology with animated live packet paths, alongside My Atlas and pinned route evidence. Work remains on `n30nex-test`. Server #192 and docs #7 stay draft, with no new review requests. There is no new diff --git a/app_documentation/post-140-roadmap.md b/app_documentation/post-140-roadmap.md index a5bbfbe..ff4d05f 100644 --- a/app_documentation/post-140-roadmap.md +++ b/app_documentation/post-140-roadmap.md @@ -1,31 +1,27 @@ # Beacon experimental roadmap: 2.0 baseline and feature parity -## Current checkpoint — 1 October 2026 - -The deployed UI pass is web `462d49e`: My Atlas leads desktop and mobile navigation. -Topology has one region selector and a compact toolbar, an inspector that opens on -selection/search, a mobile details sheet and expandable Display/Live activity panels. -The camera now starts above the mesh; Fit centres the projected bounds and drag -defaults to pan, with Shift-drag rotating. All paths and live traffic remain available. -Route-window loading keeps the existing graph visible while the new data arrives. -There are no backend, database, configuration or retention changes. - -Mesh Pulse includes Atlas, route evidence and the connected-layout/camera patch -`117ee99`. Regions and nodes are placed using actual connections; larger groups get -more space. All loaded paths remain visible by default. Region focus, isolation, -pan/orbit/zoom, top view and full screen work alongside live packet animation. -The topology now includes resolved saved routes, with 15-minute (default), one-hour -and 24-hour windows, plus 60-second live trails. Static drawing is cached separately. - -The 2.0 baseline has landed upstream. The latest compatibility check found source -conflicts with dev, including the separately prepared 2.0 integration; its earlier -clean-merge receipt is no longer current. This focused frontend patch applies cleanly -to that prepared checkout and still trial-merges cleanly with main. Refreshing the -overall integration and choosing a fresh-database/history cutover remain separate. -The live experiment keeps its legacy database and retained history. -[Exact preview, validation and recovery](n30nex-test-preview.md). - -## Maintainer direction — 30 September 2026 +## Current checkpoint — 2 October 2026 + +Prepare for stable 2.0 first; My Atlas and Topology are held for later acceptance. +The new upstream hourly-rollup commits are incorporated into the experimental +branches: server `ae328cb6` on `98006a9`, web `9d3f9585` on `39e921c`. +Upstream Analytics, schema, generated SQL, rollup workers and dependencies remain +unchanged by our feature patches. Atlas is one focused commit, Topology is the next; +optional scope metadata and exact-route evidence are separate backend packages. +[Merge sequence, exact sources and validation](post-20-integration.md). + +Native PostgreSQL/build/vet checks and 1,180 web tests pass. Desktop/French phone +browser checks retain all paths, animated traffic, camera controls and isolation. +My Atlas #97 is refreshed against current dev and held as draft after 2.0. No review +requests or new Topology PR were made. Daily checks remain paused. + +The Pi still runs `644960e4` / `462d49e8`. Its database, retention, source offer and +rollback are unchanged. A stable tag and a separate database/history decision are +still required before deploying the prepared pair. The current request authorizes +preparation, not a clean-start reset. Subsequent parity features remain queued. +[Exact preview and recovery](n30nex-test-preview.md). + +## Historical maintainer direction — 30 September 2026 The contributor shared a maintainer discussion whose final proposed target is **2.0.0**, superseding the earlier 1.7.0 suggestion and 1.4.0 planning target. diff --git a/app_documentation/post-20-integration.md b/app_documentation/post-20-integration.md new file mode 100644 index 0000000..39c5c6c --- /dev/null +++ b/app_documentation/post-20-integration.md @@ -0,0 +1,94 @@ +# My Atlas and Topology after Beacon 2.0 + +Prepared on 2 October 2026 against server `98006a93645c9f8b09d2ea441a5776eecb9add02` +and web `39e921c277a8a1c8ba79db04131a6cc8fa27649d`. These are the published dev +commits introducing hourly rollups and empty-region handling. A stable 2.0 tag is +not published at this checkpoint. Recheck the final release refs before acceptance. + +My Atlas and Topology remain post-release experiments. No release tag, production +deployment, database reset, retention change or review request accompanies this +preparation. Existing experimental PRs stay draft. The daily compatibility job +remains paused; this was a manually requested preparation pass. + +## Small merge sequence + +| Change | Focused commit | Parent and scope | +|---|---|---| +| My Atlas | [`d1eb973`](https://github.com/n30nex/beacon-web-contributions/commit/d1eb973b18d34ff4ff712992d9f5a63297431646) | Web dev `39e921c`; saved full-key cards, graphical samples, lazy details, existing inspection callbacks and English/French labels. No backend prerequisite. | +| Topology | [`8beb435`](https://github.com/n30nex/beacon-web-contributions/commit/8beb43598fc690f649a564f0efbc0dedb6af4e9d) | Apply after Atlas. Adds the canvas, bounded snapshot/live traffic, region isolation, camera controls and lazy scope metadata. | +| Optional Topology scope context | [`6ad59d3`](https://github.com/n30nex/beacon-server-contributions/commit/6ad59d31291b8abe6a7d2bd7a055de05dbeb319d) | Server dev `98006a9`; exposes cached discovered MeshMapper catalogues and the already-selected node-list default scope. No schema or query changes. | +| Exact route evidence, server | [`4fda8ed`](https://github.com/n30nex/beacon-server-contributions/commit/4fda8ed993c7fb130075c7e82fcc20c02e1fc321) | Follows the scope-context commit. Pins path bytes/width in evidence queries and cursors. Independent of either new page. | +| Exact route evidence, web | [`bc6f5c0`](https://github.com/n30nex/beacon-web-contributions/commit/bc6f5c045e1552d62c3f3ca17fbcbdf1fd13bf9e) | Follows Topology; uses upstream's combined Route Detail panel. Apply with server evidence support. | + +The web feature stack lives on `codex/post-20-my-atlas`, `codex/post-20-topology` +and `codex/post-20-route-evidence` in the contribution fork. The server stack uses +`codex/post-20-topology-support` and `codex/post-20-route-evidence`. Review each +commit against its named parent rather than importing historical preview changes. +Atlas can land alone; Topology is deliberately stacked on its shared navigation +and cancellable node reads. Neither page depends on the exact-route evidence patch. + +[My Atlas PR #97](https://github.com/MeshCore-Beacon/beacon-web/pull/97) is refreshed +to `91a9541cd6fe27f76996c8f7e65fb07698783a98` and held as draft after 2.0. Its tree +is identical to the focused Atlas commit; the older PR ancestry is preserved without +a force push. There is no new Topology PR or reviewer request. + +## Compatibility boundaries + +- The prepared server's migrations, SQL queries, generated SQL, rollup workers, + caches and retention configuration are identical to the named upstream baseline. +- The prepared web leaves upstream Analytics, region hooks, WebSocket manager and + dependency manifests unchanged. Pages are lazy-loaded, use existing inspection + panels, and do not add a rendering dependency. +- Atlas preserves `beacon-my-atlas-v1` browser storage and re-resolves an old server + ID by the full public key after a database reset. My Atlas is first on desktop + and mobile. Its 24h/3d displays describe a bounded raw-report sample, not durable + rollups or a guarantee that every deployment retains three days of raw data. +- Topology uses existing node and saved-route endpoints. An empty selected region + does not fall back to scanning global routes. Missing scope-catalogue support + shows unavailable metadata while the graph and live traffic remain usable. +- Regional MeshMapper counts never create links or prove per-node scope membership. + Only recorded neighbours and adjacent unambiguous path identities create edges. + +## Combined experimental branches and validation + +The combined `n30nex-test` heads are server `ae328cb6af31211b5935702853f09ad055d42546` +and web `9d3f95859255f4646799f7251afdeef26241364b`. Both contain their published dev +and main ancestors. The focused stacks reproduce exactly the combined source trees: +server `b7fc6bc81cb28e79e685dcb653632e10c769ed6a`, web +`26801ac745916a3fb85085cf917b4368d1683b7e`. + +- Native ARM64 server build, vet and full Go suite passed: 715 top-level tests, + including real PostgreSQL baseline/refusal, rollup, route-index and catalogue + checks. The two separately opted-in backup/export tests were skipped. +- Native Node 24.15.0 web build, lint and all 1,180 tests across 137 files passed. +- Atlas alone builds and passes 36 focused tests; the Atlas/Topology stack builds + and passes 14 focused Topology tests. The full prepared web also passed 77 focused + Windows tests before the complete native run. +- Chromium checks used the exact native-built assets with recorded topology and + WebSocket fixtures, plus a response matching the new rollup contract. They covered + desktop and French phone layouts, keyboard focus, region isolation, all paths, + pause/resume, full screen, reduced motion and 160 simultaneous live reports. + The animation sample recorded zero static redraws and no page errors. This is + fixture validation, not production-load proof or physical Safari testing. +- The dedicated validation database was created and removed. All 24 pre-existing + Pi containers retained their IDs, start times and restart counts. + +Local evidence is `F:/Beacon/evidence/release-20-prep-20261002/`. The source-tested +commits were server `4ef2712` and web `26edf55`; subsequent history-only merges have +identical Git trees. The paused web merge remains intact in its original checkout, +with an exported recovery patch and a separate clean successor. + +## Preview and release handoff + +The running Pi preview remains server `644960e4` / web `462d49e8`, with its existing +1.x migration ledger and history. **The prepared branches are ahead of deployment.** +Do not deploy the flattened baseline onto that ledger or relabel its source offer. +The new frontend also expects the new rollup API, so publish the compatible pair +together after the database/history decision. + +Keep the existing source archives and rollback described in +[the preview record](n30nex-test-preview.md). After 2.0 is tagged, compare its exact +server/web heads with the bases above, refresh only affected packages, rerun their +checks, and obtain a separate fresh-database/history cutover decision. Maintainers +retain stable merges, release tags and the production-host switch. The earlier +1.4 handoff remains historical and is not a current 2.0 deployment instruction. From 45832ec2cdc37d7f1efd6d945eac99f62abbc5f4 Mon Sep 17 00:00:00 2001 From: n30nex Date: Fri, 2 Oct 2026 11:23:34 -0400 Subject: [PATCH 54/69] docs: reconcile current status and remaining CoreScope parity --- app_documentation/post-140-roadmap.md | 350 +++++++++++--------------- 1 file changed, 148 insertions(+), 202 deletions(-) diff --git a/app_documentation/post-140-roadmap.md b/app_documentation/post-140-roadmap.md index ff4d05f..b2312b9 100644 --- a/app_documentation/post-140-roadmap.md +++ b/app_documentation/post-140-roadmap.md @@ -1,205 +1,151 @@ -# Beacon experimental roadmap: 2.0 baseline and feature parity - -## Current checkpoint — 2 October 2026 - -Prepare for stable 2.0 first; My Atlas and Topology are held for later acceptance. -The new upstream hourly-rollup commits are incorporated into the experimental -branches: server `ae328cb6` on `98006a9`, web `9d3f9585` on `39e921c`. -Upstream Analytics, schema, generated SQL, rollup workers and dependencies remain -unchanged by our feature patches. Atlas is one focused commit, Topology is the next; -optional scope metadata and exact-route evidence are separate backend packages. -[Merge sequence, exact sources and validation](post-20-integration.md). - -Native PostgreSQL/build/vet checks and 1,180 web tests pass. Desktop/French phone -browser checks retain all paths, animated traffic, camera controls and isolation. -My Atlas #97 is refreshed against current dev and held as draft after 2.0. No review -requests or new Topology PR were made. Daily checks remain paused. - -The Pi still runs `644960e4` / `462d49e8`. Its database, retention, source offer and -rollback are unchanged. A stable tag and a separate database/history decision are -still required before deploying the prepared pair. The current request authorizes -preparation, not a clean-start reset. Subsequent parity features remain queued. -[Exact preview and recovery](n30nex-test-preview.md). - -## Historical maintainer direction — 30 September 2026 - -The contributor shared a maintainer discussion whose final proposed target is -**2.0.0**, superseding the earlier 1.7.0 suggestion and 1.4.0 planning target. -This is planning evidence, not a published release or permission to reset data. -At this check, upstream server dev remains `da6de40b`, web dev `e2d272e0`, and -docs main `e671ee45`; the announced migration reset has not landed in those refs. -The existing 1.4 records below and elsewhere remain dated validation checkpoints. - -The proposed version policy is shared **major/minor** versions for server and web, -with independent patch levels. Medium/large features advance the minor version, -fixes advance the patch version, and major changes or a deliberate new baseline -advance the major version. There is no automatic major bump after a fixed number -of minor releases. Maintainers choose release versions and tags. - -The maintainer plans to flatten historical migrations and require a clean start. -Once that work lands, inspect its actual startup/upgrade contract and validate the -new baseline on a separate disposable database. Preserve the current Pi database, -configuration, source and rollback. Do not apply a flattened migration history to -the existing database or reset it automatically. A fresh preview/production switch -needs a separate cutover decision after the candidate is concrete and tested. -Any retained history/import requirements must be decided explicitly; a new empty -database does not contain the old 30-day summaries. - -Baseline compatibility and recovery validation still gate database-dependent -node/trace work. The contributor has since prioritized the frontend topology phase below. Atlas stays enabled in the experiment; its inclusion in the stable -2.0 release has not been decided. The existing daily job checks and reports only; -it does not automatically rebase, push, deploy or reset databases. - -A follow-up maintainer discussion proposes **24-hour raw packets on dev** and -**7-day raw packets in production**, to keep development data and migrations -smaller. These are environment-specific proposals, not applied settings. They do -not authorize shortening the Pi preview's existing 72-hour raw retention or -purging its data. The previously agreed 30-day summary policy has not been changed -by this discussion. Validate the eventual configuration per deployment; offer raw -history windows that match that retention, with longer windows only where durable -summaries support them. The old fixed 24h/3d decision must be reconciled with this -proposed deployment split rather than applied globally. - -The maintainer also reports that dev now uses rotational storage. Treat that as -reported environment context, not a measured cause of any latency change; compare -performance with storage, data volume and workload identified. Keep the source/ -changelog as the release audit trail. When a new baseline is accepted, distinguish -its included work from later experimental changes while retaining older history -and matching source downloads. - -## Experimental delivery scope - -Approved direction, 30 September 2026: deliver useful CoreScope feature parity in -small validated phases, then extend Beacon's regional and evidence-based analysis. -Development work lives on the explicitly requested `n30nex-test` branches in server, -web and docs, with focused pull requests and the Pi preview as validation. -The branch is experimental: keep existing PRs in draft and do not request or ping -for review until the contributor asks. Daily upstream dev/main compatibility checks -are scheduled; inspect dirty work first, use isolated trial merges, and report only -new changes, conflicts or required decisions. Routine checks do not push, deploy -or merge changes automatically. Version assignments remain with maintainers. The -earlier 1.4.0 handoff is historical and separate from this development queue. -My Atlas is explicitly included in the experimental branch and Pi preview. - -[Current experimental build, validation and recovery](n30nex-test-preview.md). - -## CartoLite-inspired preview phase — 1 October 2026 - -The contributor moved a graphical, animated topology experiment ahead of the -node/trace dashboards. Work stays on `n30nex-test`, including Atlas, without review -pings. This phase reuses existing node, neighbour and live packet data; it does not -need the proposed 2.0 database reset. Baseline compatibility remains a separate gate. - -Reference audit: [CartoLite](https://github.com/n30nex/CartoLite/tree/f2b4bdff314094c22749819ddc6817d545aa0fa0), -particularly [Netgraph](https://github.com/n30nex/CartoLite/blob/f2b4bdff314094c22749819ddc6817d545aa0fa0/docs/netgraph.md), -`netgraph/layout.ts`, `quality.ts`, `follow.ts`, `nodeInspector.ts` and `routeFocus.ts`. -CartoLite currently uses layered 2D canvases. Beacon adds a real perspective camera -and 3D node positions with native canvas drawing, without a new rendering dependency. -The implementation is original; CartoLite is a design reference, not an embedded app. - -| Priority | Useful CartoLite idea | Beacon adaptation and evidence boundary | State | -|---|---|---|---| -| Now | Stable area packing and transient packet overlays | Mesh Pulse: deterministic IATA clusters, orbit/zoom, node shapes, packet-type colours; preserve unknown hops instead of drawing invented links | Delivered on the experimental Pi preview | -| Now | Finder and neighbour spotlight | Search loaded full identities, highlight existing neighbour links, inspect nodes/packets/reporting observers through Beacon panels | Included in this phase | -| Now | Compact controls and bounded drawing work | Lazy tab, 20,000 nodes, 60,000 recent routes, 100,000 links/live segments, 10,000 reports and 512 animations; static ink cache, 30 fps ceiling, reduced motion, pause/background cleanup, English/French | Included in connected-layout patch | -| Now | Regional context | Differentiate cross-IATA links and matching advertised default scopes; show cached MeshMapper catalogue counts/freshness separately | Included in this phase | -| Now | Connected layout and regional camera | Pull linked regions together, separate islands, spread connected nodes, preserve all paths, focus/isolate regions and fit/pan/orbit/zoom; full screen and shared focus URLs | Included in connected-layout patch | -| Now | Simpler navigation and controls | My Atlas first, one region picker, compact summary, on-demand inspector/activity, accessible mobile sheet, above-mesh camera and centred Fit | Included in UI pass `462d49e` | -| Now | Observed route coverage | Load adjacent resolved route segments for a selected window; keep missing-hop gaps, distinguish these from neighbour records and live reports | Included in connected-layout patch | -| Next | Constrained live follow | Follow a selected node/observer/region without camera jumps; retain explicit exit and keyboard return | Planned | -| Next | Display preferences | Save camera/detail preferences and add measured adaptive quality; static ink caching and label collision handling are implemented | Planned; measure before adding rendering complexity | -| Later | Retained traffic replay | Bounded server cursor/window, visible gaps and retention, one time controller shared by map/topology | Requires retained-evidence API contract | -| Later | Cross-view selection and discovery | Shared node/route focus between Atlas, topology, map and investigation; bounded global search | Align with phases 3–4 | -| Optional | Sound and exhibition views | Opt-in sound only after accessibility/performance feedback; no automatic audio or copied decorative Labs assets | Deferred | - -[MeshMapper Scopes API](https://wiki.meshmapper.net/scopes-api/) was rechecked. -It is public, uses a five-minute cache/ETag and a 60 requests/minute/IP limit. It -returns names and regional counts, **not per-repeater identities**. Keep Beacon's -existing configured-source importer, conditional requests and manual fallback. -A read-only `/scope-catalogues` endpoint exposes its immutable cached metadata; -opening a graph never initiates upstream HTTP or packet-table scans. A missing, -failed, disabled or stale catalogue must not create or remove neighbour evidence. -Case-sensitive scope names and monitored zero-count entries remain intact. - -The [Zones API](https://wiki.meshmapper.net/zones-api/) lists published regional -URLs by country and serves nullable GeoJSON boundaries, cached hourly with ETags. -Use that discovery contract for a future configured-source expansion and map link; -keep geometric boundary crossings separate from receiving-IATA changes. - -Shared advertised defaults are context, not proof of forwarding. An IATA group is -based on the node's latest hearing region, not geographic coordinates. Packet -colours follow high-confidence adjacent resolved hops; animation speed is illustrative. -The existing Pi importer is configured for YOW. Other IATAs must say their catalogue -is unavailable until a published source is deliberately configured; do not invent -URLs or assign a group catalogue's counts to every member region. +# Beacon roadmap: 2.0 integration and CoreScope parity + +## Current status — checked 2 October 2026 + +| Surface | Verified state | +|---|---| +| Upstream server | `dev` **98006a9** includes the 2.0 baseline and hourly rollups; `main` remains **201cd9e**. Latest published release is still **v1.6.0**. | +| Upstream web | `dev` **39e921c** consumes hourly rollups and handles empty regions; `main` remains **5ac36ce**. Latest published release is still **v1.3.0**. | +| Upstream docs | `main` **24d2e5f**; our experimental roadmap remains in draft PR #7. | +| Prepared experiment | `n30nex-test`: server **ae328cb6**, web **9d3f9585**. Both incorporate the named upstream dev/main heads. Server #192 and My Atlas #97 are draft, mergeable and pass their published-head CI; web CodeQL is skipped. | +| Pi preview | Still server **644960e4** / web **462d49e8**, confirmed by the public source offer. Its legacy database, 72h raw/30d summary retention and rollback are unchanged. | +| Official sites | `live.meshcore.ca` serves Beacon and displays **2.0.0**. `dev.meshcore.ca` also displays **2.0.0**, with a development-only banner linking everyday users to live. Backend revisions and operator rollout acceptance were not verified by this read-only browser check. | +| Release boundary | **No stable 2.0 GitHub release/tag yet.** A frontend version label is not a published release. My Atlas and Topology remain held for post-2.0 acceptance. | + +The prepared code passed native server build/vet/PostgreSQL validation and web +build/lint/**1,180 tests**. Two opt-in backup/export tests were skipped. Browser +checks covered desktop, French phone layouts, region isolation, all loaded paths +and 160 animated packet reports. This roadmap-only refresh did not rerun those +unchanged-code checks or deploy anything. Daily compatibility automation remains +**paused**, and no review request is authorized. + +[Exact post-2.0 packages and merge order](post-20-integration.md) · +[Actual preview and rollback](n30nex-test-preview.md) · +[Pi source offer](https://canadaverse.org/beacon-dev/source.html). + +## Comparison baseline and scope + +This inventory compares Beacon's published dev code plus the prepared experiment +with [CoreScope master `093e320`](https://github.com/Kpa-clawbot/CoreScope/tree/093e320c2bda99d1fef317d7fc21fc1240a8cd12), +checked on 2 October. CoreScope's latest release is +[v3.12.0](https://github.com/Kpa-clawbot/CoreScope/releases/tag/v3.12.0); +master contains later fixes, so this is a source-backed workflow comparison, +not a claim that every reference feature is deployed on every CoreScope instance. +The old `live.meshcore.ca` CoreScope screenshots are historical references now. + +Parity means useful user workflows with honest counting and bounded cost. It does +not require matching CoreScope's architecture, copying every tab, or reproducing +its performance claims on different storage and workloads. The remaining feature +list is separated from release acceptance and production-load qualification. + +## What is already covered + +| Capability | Beacon state | Remaining boundary | +|---|---|---| +| Live packet feed and decoding | Upstream: filters, byte/payload inspection, per-observer receptions, signal values, shared links and mapped packet paths | A graphical reception timeline can still improve investigation. | +| Live map and neighbour graph | Upstream: live packet animation, node/role controls, map links, neighbour view and IATA borders | Live animation does not provide historical replay. | +| Channels and scope context | Upstream: decrypted messages, Public/hashtag keys, regional channel/scope filters and MeshMapper catalogue discovery/import | Per-channel historical analytics are still missing; unknown names cannot be inferred from a short hash alone. | +| Observer monitoring | Upstream: unified sidebar/dashboard, activity/type/signal charts, telemetry and comparison | Reach by confirmed node/hop, fleet comparisons and timing interpretation need deeper work. | +| Network analytics | Upstream: traffic, signal, path/hop/hash-width distributions, scopes, talkers, clock drift and neighbour graph; hourly summaries survive raw expiry | These aggregate views do not replace node or channel dashboards. | +| Navigation and localization | Upstream: entity inspection, shareable routes/packets/observers, mobile layouts and English/French | Global search, saved filter presets and cross-page Atlas filtering remain. | +| My Atlas | Implemented in the Pi experiment and prepared as a focused post-2.0 feature; PR #97 is refreshed, draft and clean | Acceptance after 2.0 remains. Cards use up to 200 retained origin-key reports, not complete node totals. | +| 3D Topology | Implemented in the Pi experiment; a focused prepared commit follows Atlas | Acceptance after 2.0 remains. All paths, live animation, camera controls, region focus/isolation and lazy scope context are implemented. Follow mode, saved display preferences and replay are later work. | +| Exact route evidence | Upstream already fixes saved-prefix refresh. The experiment adds pinned bytes/width, cursors and shared windows in separate patches | Accept the paired server/web evidence patches separately; neither new page depends on them. | + +## Remaining CoreScope parity + +| Area | What is left | Reference and current Beacon limit | +|---|---|---| +| **Node and repeater dashboards** | Full-key activity history, packet-type mix, signal/hop distributions, hearing coverage, activity heatmaps, peer relationships and comparable repeater/fleet metrics | [Node analytics source](https://github.com/Kpa-clawbot/CoreScope/blob/093e320c2bda99d1fef317d7fc21fc1240a8cd12/public/node-analytics.js), [fleet/relay analysis](https://github.com/Kpa-clawbot/CoreScope/blob/093e320c2bda99d1fef317d7fc21fc1240a8cd12/public/analytics.js). Beacon has node details/recent reports and Atlas samples, but no equivalent complete dashboard. | +| **Packet/trace chronology and observer reach** | A graphical reception timeline, observer-to-observer spread, confirmed reach by hop/node, and a consistent return path into packets, routes and map | [CoreScope tracing overview](https://github.com/Kpa-clawbot/CoreScope/blob/093e320c2bda99d1fef317d7fc21fc1240a8cd12/README.md#and-more). Beacon already shows first/last times, per-observer signal and trace hop chains; the gap is richer analysis, not basic packet inspection. | +| **Global search and saved mesh filters** | Ctrl+K-style search across nodes, observers, packets and channels; use Atlas selections across views; saved filter/layout presets | [Global search and favorites](https://github.com/Kpa-clawbot/CoreScope/blob/093e320c2bda99d1fef317d7fc21fc1240a8cd12/public/app.js). Beacon searches individual lists and saves cards, but has no shared search or site-wide saved-node filter. | +| **Channel analytics** | Messages over time, channel comparisons, per-channel senders and hearing context, with explicit key/history availability | [Analytics guide](https://github.com/Kpa-clawbot/CoreScope/blob/093e320c2bda99d1fef317d7fc21fc1240a8cd12/docs/user-guide/analytics.md#channels). The decoded message viewer and general talker statistics already exist. | +| **Hash and prefix tools** | Collision/usage matrix, role-aware prefix-width analysis and a prefix checker with explicit ambiguity | [Hash and prefix analysis](https://github.com/Kpa-clawbot/CoreScope/blob/093e320c2bda99d1fef317d7fc21fc1240a8cd12/public/analytics.js). Beacon already displays width distributions and ambiguous candidates; it lacks the dedicated tools. | +| **Distance and route patterns** | Valid-coordinate hop/path distances, signal-versus-distance views, common subpath rankings and evidence-linked route alternatives/inspection | [Distance and route-pattern analysis](https://github.com/Kpa-clawbot/CoreScope/blob/093e320c2bda99d1fef317d7fc21fc1240a8cd12/public/analytics.js). Existing route search, detail and 3D layout do not supply geographic-distance analytics. | +| **Historical replay** | Play/pause/seek, stepping and speed controls over retained observations; one time controller for map and Topology, with visible gaps | [Live/VCR guide](https://github.com/Kpa-clawbot/CoreScope/blob/093e320c2bda99d1fef317d7fc21fc1240a8cd12/docs/user-guide/live.md#vcr-mode). Beacon's live pause and static route-history windows are not replay. Hourly rollups cannot recreate expired packet paths. | +| **Geographic area filtering** | Filter nodes and their attributed traffic by advertised location/polygon across views, separately from receiving-IATA groups | [Area filter](https://github.com/Kpa-clawbot/CoreScope/blob/093e320c2bda99d1fef317d7fc21fc1240a8cd12/docs/user-guide/area-filter.md). Beacon's IATA groups, boundary overlay and optional foreign-node classification do not provide this complete workflow. | + +Smaller parity items remain below those analysis workflows: node/channel QR sharing, +more table/layout preferences, an in-browser theme editor with import/export, and +operator-facing performance diagnostics. Existing theme selection and deployment +configuration are not a theme editor. See +[CoreScope customization](https://github.com/Kpa-clawbot/CoreScope/blob/093e320c2bda99d1fef317d7fc21fc1240a8cd12/docs/user-guide/customization.md), +[channel QR](https://github.com/Kpa-clawbot/CoreScope/blob/093e320c2bda99d1fef317d7fc21fc1240a8cd12/public/channel-qr.js) +and [performance UI](https://github.com/Kpa-clawbot/CoreScope/blob/093e320c2bda99d1fef317d7fc21fc1240a8cd12/public/perf.js). +Audio/Lab and decorative exhibition modes are optional, not blockers for the core +analysis roadmap. Account synchronization/MeshMapper login is a separate ownership +and authentication decision, not a prerequisite for browser-local Atlas. ## Delivery order -| Phase | Deliverable | Completion evidence | Status | -|---|---|---|---| -| Separate gate — upstream 2.0 baseline | Inspect synchronized versioning and the flattened migration contract when published | Validate an isolated fresh database, API/feature compatibility, available history and recoverable cutover; preserve the running Pi database | Landed during this phase; isolated integration and fresh-database checks pass, cutover/history decision pending | -| 1 — correctness | Saved-route hash-width consistency, then retention-aware time controls and remaining French/mobile/accessibility fixes | Route identity survives representation changes; evidence pagination and shared windows do not silently change path; raw and summary periods match available data | Route fix delivered to the experimental Pi preview; remaining controls/polish are separate; server PR #192 stays draft | -| 2 — experimental Atlas | My Atlas saved-node monitoring | Carry the feature from web PR #97 into the experiment; saved identities/order survive; compact cards, expandable Heard by/statistics and existing entity links work in English/French on desktop/phone | Delivered in n30nex-test; stable-release inclusion remains undecided | -| 3 — node and trace investigation | Node dashboard, activity/type/signal/hop analysis, trace reception timeline and complete return navigation | Separate attributed node traffic from possible prefix matches; packet → route → node/observer → map links preserve selection, filters and Back | Queued after baseline validation | -| 4 — find and compare | Bounded global entity search, saved views/filters, Atlas-node filters, channel activity and hearing context | Search/paging/share links agree; channel key/history availability is explicit; comparisons use aligned windows | Queued | -| 5 — network structure | Observed route segments/alternatives, topology, distance, hash ambiguity and prefix/path inspection | Count evidence at the correct grain; separate observed ambiguity from static conflicts; use valid coordinates and show unresolved hops | 3D topology and scope context delivered; distance/hash inspection remains queued | -| 6 — history and reach | Bounded retained map replay, observer reach and timing analysis, comparable fleet telemetry | Replay preserves ordering and retention limits; confirmed identities are separate from unresolved prefixes; timing/counter gaps are not labelled packet loss | Queued | -| Beyond parity | Regional boundary/scope crossing investigation and links between observations, discovered scopes and route changes | Each relationship links to retained evidence; distinguish reported locations from inferred paths and scope names from geography | Queued after supporting phases | - -The initial Atlas cards show a bounded sample of the latest 200 retained -origin-key reports per node. Complete node totals and longer history need an -explicit aggregate contract in phase 3; do not relabel the sample as total traffic. -Browser-local cards remain the first release. Account sync or MeshMapper login -needs a separately agreed authentication and ownership contract. - -## First implementation: saved-route evidence - -[Server #183](https://github.com/MeshCore-Beacon/beacon-server/issues/183) remains -open, with the server correction in [PR #192](https://github.com/MeshCore-Beacon/beacon-server/pull/192). The same fully resolved node chain can arrive with different hash widths, -while its saved prefix metadata currently stays at the first representation. -That can omit newer exact-path observations from route evidence. - -Keep the stable IATA/node-chain identity. Define how the latest representation is -updated and how an already-open evidence page retains its selected representation. -Cover successive 1/2/3-byte paths, unchanged repeats, malformed/missing metadata, -ambiguous identities, cursor precision and shared-window behavior. Preserve exact -bytes and the existing bounded observation index; do not replace the query with -a broad short-prefix or raw-history scan. Avoid a schema change unless the agreed -behavior requires one. This phase must include a real PostgreSQL regression and -the exact combined Pi build before its preview is called verified. - -## Parallel priorities, scheduled deliberately - -- Operational work: remaining admin configuration scope, browser access and - import/restore, deployment-file coverage and remote/scheduled backup under - [server #60](https://github.com/MeshCore-Beacon/beacon-server/issues/60) and - [#72](https://github.com/MeshCore-Beacon/beacon-server/issues/72). -- MQTT timeout attribution under - [#116](https://github.com/MeshCore-Beacon/beacon-server/issues/116): collect - evidence on recurrence or a meaningful workload change. A healthy short sample - does not identify the historical cause. -- Complete translations under - [web #12](https://github.com/MeshCore-Beacon/beacon-web/issues/12), with touched - interface text translated in the same feature PR. -- Reconcile older open tickets against accepted code. Several observer, packet - and route tickets were implemented through the consolidation batch; their open - state alone is not a new feature gap. Close only after their full scope is checked. - -## Shared acceptance criteria - -Every new view defines its counting unit, effective period, available history, -sample size, missing data and ambiguity. Raw packets and observations cannot be -reconstructed after expiry; longer-lived summaries must state what they preserve. -Reuse the existing observer, packet, route, map and chart components. Load only the -selected data, coalesce updates, bound requests and test query plans on -representative data. New aggregation is justified by a measured query need. - -Validate source and actual running artifacts, native PostgreSQL behavior, relevant -ingestion/reconnect regressions, desktop/phone, English/French, keyboard and shared -links. Publish matching source/changelog and preserve rollback without overwriting -new traffic. Compare CPU/memory/storage and latency against the preceding build; -a fixture or short Pi sample does not certify production capacity. - -Maintainers control merges, version numbers, stable artifacts and the production -switch. `live.meshcore.ca` is the planned production destination; `dev.meshcore.ca` -remains development-only. The Canadaverse Pi remains the contribution preview. +| Phase | Next package | Completion gate | +|---|---|---| +| **0 — stable baseline** | Recheck the actual 2.0 release refs, then decide the Pi's fresh-database/history cutover | Keep existing data and recovery; publish a compatible server/web pair only after that decision. Official hosts remain owner-controlled. | +| **1 — accept prepared work** | My Atlas, then Topology; optional scope context and exact-route evidence remain separate | Refresh only changed bases; keep the focused merge sequence, current-head checks, paired API compatibility and draft hold until review is requested. | +| **2 — node and trace investigation** | Full-key node dashboard first, then packet/trace chronology and fleet comparisons | Define origin reports versus confirmed relay evidence; share counting/window rules across cards/charts. Reuse existing rollups where their dimensions fit; measure before adding a node aggregate. | +| **3 — finding and channel analysis** | Global search, Atlas-node filters/saved views, channel activity and sender/hearing comparisons | Bounded queries, stable entity keys, aligned windows, keyboard access and share/Back behavior. | +| **4 — network tools** | Hash/prefix ambiguity, distance, repeated subpaths and route alternatives | Keep unknown identities and invalid coordinates visible; every route claim links to recorded evidence. | +| **5 — history and geography** | Retained packet replay, observer reach/timing and GPS-area filtering | Ordered bounded cursors, explicit retention/gaps, no invented paths or clock-based claims of RF propagation. | +| **Beyond parity** | Evidence-linked MeshMapper scope/boundary crossings, region changes and guided live follow | Distinguish a receiving-IATA change, advertised location, scope label and a geometric crossing; never manufacture neighbour links from catalogue counts. | + +**Recommended next new implementation after 2.0:** the node dashboard/trace phase. +It builds on the existing node inspector and Atlas instead of adding another +independent page with different counts. Replay is the largest remaining live-view +capability gap and needs its retained-evidence contract before UI implementation. +This update schedules work; it does not start new feature implementation. + +## CartoLite-derived follow-ups + +[CartoLite `f2b4bdf`](https://github.com/n30nex/CartoLite/tree/f2b4bdff314094c22749819ddc6817d545aa0fa0) +remains a design reference. Stable regional groups, actual-link placement, finder, +neighbour emphasis, region isolation, compact controls, above-mesh camera and +separate static/live rendering are already delivered experimentally. The current +limits remain 20,000 nodes, 60,000 recent routes, 100,000 links/live segments, +10,000 reports and 512 simultaneous animations, with background cleanup and +reduced-motion support. These are ceilings, not claims of a complete network view. + +Next Topology-specific refinements are constrained live follow, saved camera/display +preferences and measured adaptive quality. Cross-view selection belongs with phase +3; replay belongs with phase 5. Scope metadata stays optional and lazy. Upstream +now discovers regional scope/channel sources through MeshMapper zone lists; that +is already implemented and must not remain listed as a future importer feature. +The unchanged legacy Pi still has its recorded YOW importer configuration. + +## Issue reconciliation and operational work + +- [Server #183](https://github.com/MeshCore-Beacon/beacon-server/issues/183) is + **closed**: upstream `6cd03e7` refreshes saved-route prefix metadata. The prepared + pinned-evidence changes are additional work, not a still-open prefix fix. +- [Server #116](https://github.com/MeshCore-Beacon/beacon-server/issues/116) and + [web #12](https://github.com/MeshCore-Beacon/beacon-web/issues/12) are **closed**. + Reopen investigation only for new MQTT evidence or a specific localization defect; + continue translation/accessibility checks on every new UI change. +- [Web #96](https://github.com/MeshCore-Beacon/beacon-web/issues/96) remains tied to + held My Atlas PR #97. Experimental delivery is not stable acceptance. +- [Server #60](https://github.com/MeshCore-Beacon/beacon-server/issues/60) and + [#72](https://github.com/MeshCore-Beacon/beacon-server/issues/72) remain open for + admin and local/remote backup scope. Existing protected settings/accounts/exports + are partial delivery; public admin and backup access remain disabled on our Pi. +- Sustained production-like ingest/query load, recovery/export qualification and + physical Safari checks remain operational acceptance work. Passing fixtures or + CoreScope's published benchmarks do not establish Beacon production capacity. + +## Shared acceptance rules + +Every view states its counting unit, effective time window, sample cap, missing +history and identity ambiguity. Receptions, unique packets, origins and relay +candidates are different metrics. A missing record or unsynchronized clock does +not establish packet loss or RF delay. Raw chart choices follow actual deployment +retention; durable summaries only expose dimensions that were retained. + +Reuse Beacon's inspection panels, region model, query cache, chart components and +new hourly-rollup contract. Keep bounded indexed queries, lazy loading and coalesced +refreshes; do not fetch separately per chart or add a blanket materialized-view +refresh job. Native database and relevant replay/browser checks precede deployment. +No feature patch in this queue should alter the flattened baseline migration. + +Keep source offers and rollback tied to the actual running pair, preserve unrelated +work, and retain the paused merge recovery. Prepared 2.0 branches must not be +applied automatically to the Pi's 1.x ledger. Earlier 1.4/1.7 planning is historical; +server/web major and minor versions track maintainer decisions, with independent +patches. Merges, stable tags and official-host operations remain maintainer-controlled. From e0beea4506030a4f72dbb43ede82d424f66f90a1 Mon Sep 17 00:00:00 2001 From: n30nex Date: Fri, 2 Oct 2026 21:59:49 -0400 Subject: [PATCH 55/69] docs: record private collector service and deployed 2.1 experiment --- app_documentation/n30nex-test-preview.md | 52 +++++++ app_documentation/post-140-roadmap.md | 71 +++++++--- app_documentation/post-20-integration.md | 52 +++++++ app_documentation/post-20-ui-telemetry.md | 161 ++++++++++++++++++++++ 4 files changed, 314 insertions(+), 22 deletions(-) create mode 100644 app_documentation/post-20-ui-telemetry.md diff --git a/app_documentation/n30nex-test-preview.md b/app_documentation/n30nex-test-preview.md index de666ad..cb5c6f3 100644 --- a/app_documentation/n30nex-test-preview.md +++ b/app_documentation/n30nex-test-preview.md @@ -1,5 +1,57 @@ # Experimental n30nex-test preview +## Current: 2.1 experimental preview — 3 October 2026 UTC + +The user approved **private collector repository + separate intake service**, fresh +preview history with the old database intact, and continued exclusive collector use +of the RemoteTerm radio. These direct decisions supersede older cutover/restore +holds below. RemoteTerm stays **stopped and disabled** until the user asks for it back. + +`n30nex-test` is now server **fe4c4156** / web **4b189dff**, incorporating upstream +server dev **af20beb** / web dev **0924260** and main. Both application branches and +experimental tags are published to the contribution forks. My Atlas and Topology +remain experimental; server #192 and Atlas #97 remain draft. No review ping or +upstream stable release was made; daily automation remains **PAUSED**. + +The Pi publicly runs those exact server/web revisions. Core uses the new +`beacon_post21_preview_20261002` database (001 baseline); the old **beacon_dev** +with its 045 ledger is intact. The old dump was restore-tested and its SHA-256 +verified on/off Pi. Raw/summary retention stays 72 hours/30 days. + +Beacon Collector intake **84b0d22** runs in its own container and dedicated +`beacon_collector_preview` database/role. Its repository is **private**, with the +existing license unchanged. Public telemetry routes go to this service; Beacon +core has no collector table, migration or package dependency. Never push the local +embedded prototype history (`1b10530` through `633a380`) into public Beacon refs. +The public server was composed from its clean parent instead. + +753 native server checks and 1,255 frontend tests passed, plus build/lint/vet, +real PostgreSQL checks, the 3,200-input replay (zero fixture drops), exact public +assets/source hashes and automatic signed HTTPS enrollment/delivery. Two opt-in +backup/export tests were skipped. The replay exceeded its original 20-second +shared-host budget and passed at 22.80 seconds with a 60-second allowance; this +is not a production throughput guarantee. + +Five configured repeaters have returned telemetry: Hilltop, Weaver, Royal City, +Starkey and Royal Relay. Reservoir still times out. Hilltop supplied real channel-2 +temperature/humidity/pressure as well as battery. Missing data and timing gaps stay +visible. Normal polling keeps 1–72-hour intervals and congestion checks; no test +bypass is shipped. Hardware attestation and cross-client polling leases are not +implemented. The collector defaults to Canadaverse until a later Beacon release. + +Current paired rollback: `evidence/post21-final-20261003/deploy-preview.py rollback` +on Pi; checkpoint **post21-cutover-20261003T014506Z**. It restores the retained +legacy container/config/frontend while preserving the new core and telemetry DBs. +The private DB dump is **post21-fresh-20261003T002912Z**. Old rollback commands are +historical and must not be applied directly against this pair. + +Canonical state: `planning/experimental-n30nex-test.json`; public source offer: +https://canadaverse.org/beacon-dev/source.html. Final evidence is under +`evidence/post21-final-20261003`. Keep the old dirty `topology-web-2-integration` +checkout untouched. Docs changes were preserved in `docs-n30nex-test` and prepared +in `docs-telemetry-20261003` for integration. + + ## Prepared branch versus deployed preview — 2 October The prepared 2.0 branch heads are server `ae328cb6` and web `9d3f9585`, including diff --git a/app_documentation/post-140-roadmap.md b/app_documentation/post-140-roadmap.md index b2312b9..6fc629a 100644 --- a/app_documentation/post-140-roadmap.md +++ b/app_documentation/post-140-roadmap.md @@ -1,27 +1,54 @@ # Beacon roadmap: 2.0 integration and CoreScope parity -## Current status — checked 2 October 2026 - -| Surface | Verified state | +## Current delivery — 3 October 2026 UTC + +The [Pi preview](https://canadaverse.org/beacon-dev/) now runs experimental +**2.1.0-n30nex.1**, server **fe4c4156** / web **4b189dff**, including the checked +upstream **af20beb / 0924260**. The 2.0.1 patch milestone and 2.1.0 feature tags +are published on contribution forks; they are not upstream stable releases. + +Delivered in the preview: + +- Topology left-pan/right-orbit/wheel zoom, scene-anchored labels, route-window + refresh, viewport-sized Live Activity, region isolation and fullscreen. +- Full node pages with bounded report summaries, observer sparklines, visible live + map icons with reception glow, and compact Routes/Traces. +- My Atlas battery and environmental telemetry via the **separate Beacon Collector + service/database**. The collector repository is private; its license is unchanged. + Automatic enrollment does not require operator approval. +- Fresh 2.0 baseline history, preserving the old database intact and its verified + on/off-Pi recovery dump. 72-hour raw/30-day summary retention remains. + +Verified: **753 server tests, 1,255 web tests**, native PostgreSQL/build/lint/vet, +3,200 replay inputs with zero fixture drops, 30 public assets and both corresponding +source archives, signed public HTTPS delivery, duplicate rejection and invalid +signature rejection. Initial 20-second replay timing failed under concurrent load; +the retained 60-second-budget run passed in 22.80 seconds. Two optional backup tests +remain skipped. Browser checks cover live traffic, camera controls, route refresh, +region isolation, fullscreen, full-node details and French phone containment. + +Five of the six selected solar repeaters have returned readings. Reservoir has not. +Hilltop supplied channel-2 temperature, humidity and pressure plus battery. Graphs +require later successful readings and retain gaps; no values are fabricated. The +leased radio stays on the normal hourly collector schedule and RemoteTerm remains +suspended at the user's request. + +Rollback is `evidence/post21-final-20261003/deploy-preview.py rollback`, checkpoint +`post21-cutover-20261003T014506Z`. The core and telemetry databases remain separate. +Review requests and the daily compatibility automation remain on hold. Official +production deployment and stable version selection belong to the maintainers. + +## Remaining parity and next phases + +| Area | Remaining work | |---|---| -| Upstream server | `dev` **98006a9** includes the 2.0 baseline and hourly rollups; `main` remains **201cd9e**. Latest published release is still **v1.6.0**. | -| Upstream web | `dev` **39e921c** consumes hourly rollups and handles empty regions; `main` remains **5ac36ce**. Latest published release is still **v1.3.0**. | -| Upstream docs | `main` **24d2e5f**; our experimental roadmap remains in draft PR #7. | -| Prepared experiment | `n30nex-test`: server **ae328cb6**, web **9d3f9585**. Both incorporate the named upstream dev/main heads. Server #192 and My Atlas #97 are draft, mergeable and pass their published-head CI; web CodeQL is skipped. | -| Pi preview | Still server **644960e4** / web **462d49e8**, confirmed by the public source offer. Its legacy database, 72h raw/30d summary retention and rollback are unchanged. | -| Official sites | `live.meshcore.ca` serves Beacon and displays **2.0.0**. `dev.meshcore.ca` also displays **2.0.0**, with a development-only banner linking everyday users to live. Backend revisions and operator rollout acceptance were not verified by this read-only browser check. | -| Release boundary | **No stable 2.0 GitHub release/tag yet.** A frontend version label is not a published release. My Atlas and Topology remain held for post-2.0 acceptance. | - -The prepared code passed native server build/vet/PostgreSQL validation and web -build/lint/**1,180 tests**. Two opt-in backup/export tests were skipped. Browser -checks covered desktop, French phone layouts, region isolation, all loaded paths -and 160 animated packet reports. This roadmap-only refresh did not rerun those -unchanged-code checks or deploy anything. Daily compatibility automation remains -**paused**, and no review request is authorized. - -[Exact post-2.0 packages and merge order](post-20-integration.md) · -[Actual preview and rollback](n30nex-test-preview.md) · -[Pi source offer](https://canadaverse.org/beacon-dev/source.html). +| Node/repeater analytics | Full history beyond the bounded 200-report sample; signal/hop distributions, heatmaps, hearing coverage and comparable fleet metrics. | +| Trace investigation | Graphical reception chronology and evidence-linked observer reach; compact rows are now delivered. | +| Finding and channel analysis | Global search, saved mesh/Atlas filters, channel activity and sender/hearing comparisons. | +| Network tools | Prefix collisions/checker, geographic distance analysis, repeated subpaths and route alternatives. | +| Replay and geography | Retained packet playback/seek/speed, reach/timing analysis and GPS-area filtering. | +| Collector release readiness | Cross-client per-repeater leases, abuse/revocation controls, packaged USB/BLE onboarding, long-run reconnect/delivery tests and diagnosis of unresponsive targets. A possible Beacon 3.0 pairing remains planning. | +| Smaller optional work | QR sharing, theme/layout import/export and operator diagnostics. | ## Comparison baseline and scope @@ -56,7 +83,7 @@ list is separated from release acceptance and production-load qualification. | Area | What is left | Reference and current Beacon limit | |---|---|---| -| **Node and repeater dashboards** | Full-key activity history, packet-type mix, signal/hop distributions, hearing coverage, activity heatmaps, peer relationships and comparable repeater/fleet metrics | [Node analytics source](https://github.com/Kpa-clawbot/CoreScope/blob/093e320c2bda99d1fef317d7fc21fc1240a8cd12/public/node-analytics.js), [fleet/relay analysis](https://github.com/Kpa-clawbot/CoreScope/blob/093e320c2bda99d1fef317d7fc21fc1240a8cd12/public/analytics.js). Beacon has node details/recent reports and Atlas samples, but no equivalent complete dashboard. | +| **Node and repeater dashboards** | Full history beyond the delivered 200-report node page, signal/hop distributions, hearing coverage, activity heatmaps, peer relationships and comparable repeater/fleet metrics | [Node analytics source](https://github.com/Kpa-clawbot/CoreScope/blob/093e320c2bda99d1fef317d7fc21fc1240a8cd12/public/node-analytics.js), [fleet/relay analysis](https://github.com/Kpa-clawbot/CoreScope/blob/093e320c2bda99d1fef317d7fc21fc1240a8cd12/public/analytics.js). Beacon now has full node pages, report/type samples, telemetry cards and Atlas; complete node/fleet summaries remain. | | **Packet/trace chronology and observer reach** | A graphical reception timeline, observer-to-observer spread, confirmed reach by hop/node, and a consistent return path into packets, routes and map | [CoreScope tracing overview](https://github.com/Kpa-clawbot/CoreScope/blob/093e320c2bda99d1fef317d7fc21fc1240a8cd12/README.md#and-more). Beacon already shows first/last times, per-observer signal and trace hop chains; the gap is richer analysis, not basic packet inspection. | | **Global search and saved mesh filters** | Ctrl+K-style search across nodes, observers, packets and channels; use Atlas selections across views; saved filter/layout presets | [Global search and favorites](https://github.com/Kpa-clawbot/CoreScope/blob/093e320c2bda99d1fef317d7fc21fc1240a8cd12/public/app.js). Beacon searches individual lists and saves cards, but has no shared search or site-wide saved-node filter. | | **Channel analytics** | Messages over time, channel comparisons, per-channel senders and hearing context, with explicit key/history availability | [Analytics guide](https://github.com/Kpa-clawbot/CoreScope/blob/093e320c2bda99d1fef317d7fc21fc1240a8cd12/docs/user-guide/analytics.md#channels). The decoded message viewer and general talker statistics already exist. | diff --git a/app_documentation/post-20-integration.md b/app_documentation/post-20-integration.md index 39c5c6c..7f40229 100644 --- a/app_documentation/post-20-integration.md +++ b/app_documentation/post-20-integration.md @@ -1,5 +1,57 @@ # My Atlas and Topology after Beacon 2.0 +## Current: 2.1 experimental preview — 3 October 2026 UTC + +The user approved **private collector repository + separate intake service**, fresh +preview history with the old database intact, and continued exclusive collector use +of the RemoteTerm radio. These direct decisions supersede older cutover/restore +holds below. RemoteTerm stays **stopped and disabled** until the user asks for it back. + +`n30nex-test` is now server **fe4c4156** / web **4b189dff**, incorporating upstream +server dev **af20beb** / web dev **0924260** and main. Both application branches and +experimental tags are published to the contribution forks. My Atlas and Topology +remain experimental; server #192 and Atlas #97 remain draft. No review ping or +upstream stable release was made; daily automation remains **PAUSED**. + +The Pi publicly runs those exact server/web revisions. Core uses the new +`beacon_post21_preview_20261002` database (001 baseline); the old **beacon_dev** +with its 045 ledger is intact. The old dump was restore-tested and its SHA-256 +verified on/off Pi. Raw/summary retention stays 72 hours/30 days. + +Beacon Collector intake **84b0d22** runs in its own container and dedicated +`beacon_collector_preview` database/role. Its repository is **private**, with the +existing license unchanged. Public telemetry routes go to this service; Beacon +core has no collector table, migration or package dependency. Never push the local +embedded prototype history (`1b10530` through `633a380`) into public Beacon refs. +The public server was composed from its clean parent instead. + +753 native server checks and 1,255 frontend tests passed, plus build/lint/vet, +real PostgreSQL checks, the 3,200-input replay (zero fixture drops), exact public +assets/source hashes and automatic signed HTTPS enrollment/delivery. Two opt-in +backup/export tests were skipped. The replay exceeded its original 20-second +shared-host budget and passed at 22.80 seconds with a 60-second allowance; this +is not a production throughput guarantee. + +Five configured repeaters have returned telemetry: Hilltop, Weaver, Royal City, +Starkey and Royal Relay. Reservoir still times out. Hilltop supplied real channel-2 +temperature/humidity/pressure as well as battery. Missing data and timing gaps stay +visible. Normal polling keeps 1–72-hour intervals and congestion checks; no test +bypass is shipped. Hardware attestation and cross-client polling leases are not +implemented. The collector defaults to Canadaverse until a later Beacon release. + +Current paired rollback: `evidence/post21-final-20261003/deploy-preview.py rollback` +on Pi; checkpoint **post21-cutover-20261003T014506Z**. It restores the retained +legacy container/config/frontend while preserving the new core and telemetry DBs. +The private DB dump is **post21-fresh-20261003T002912Z**. Old rollback commands are +historical and must not be applied directly against this pair. + +Canonical state: `planning/experimental-n30nex-test.json`; public source offer: +https://canadaverse.org/beacon-dev/source.html. Final evidence is under +`evidence/post21-final-20261003`. Keep the old dirty `topology-web-2-integration` +checkout untouched. Docs changes were preserved in `docs-n30nex-test` and prepared +in `docs-telemetry-20261003` for integration. + + Prepared on 2 October 2026 against server `98006a93645c9f8b09d2ea441a5776eecb9add02` and web `39e921c277a8a1c8ba79db04131a6cc8fa27649d`. These are the published dev commits introducing hourly rollups and empty-region handling. A stable 2.0 tag is diff --git a/app_documentation/post-20-ui-telemetry.md b/app_documentation/post-20-ui-telemetry.md new file mode 100644 index 0000000..b11d3ce --- /dev/null +++ b/app_documentation/post-20-ui-telemetry.md @@ -0,0 +1,161 @@ +# Post-2.0 UI and node telemetry batch + +Requested 2 October 2026. Work stays experimental on `n30nex-test`, based on +upstream server `af20beb` / web `0924260` plus Atlas, Topology and exact-route +evidence. Review requests and the daily automation remain on hold. + +## Versioned delivery + +- **2.0.1 fixes**: Topology left-drag pan/right-drag orbit/wheel zoom, stable + region labels, route-window refresh, one-screen layout and retained fullscreen; + remove drag-mode buttons and explanatory sections; compact Routes and Traces. +- **2.1.0 features**: full node detail page with existing-data graphs; observer + card sparklines; Atlas telemetry where a full-key-linked observer already + provides it; visible node icons during live map traffic with reception glow. +- Use annotated **`v2.0.1-n30nex.1`** and **`v2.1.0-n30nex.1`** milestone tags on + contribution forks after validation. These are experimental prerelease tags, + not upstream stable releases. Bigger user-facing features advance 2.x; fixes + advance 2.0.x. Server/web major-minor alignment remains the release policy. + +## Requested changes and acceptance + +| Area | Work | Required proof | +|---|---|---| +| Topology camera | Left mouse pans, right mouse changes the 3D angle, wheel zooms; remove mode buttons; retain touch/keyboard and fullscreen | Mouse buttons, context-menu suppression only on canvas, touch/pinch, keyboard and region focus/isolation | +| Region labels | Anchor labels to projected scene positions; fade/shrink or omit collisions instead of moving them above the scene | Dense mesh and zoomed-out screenshots, hit targets follow visible labels | +| Route window | Fetch fresh route data on a changed window and show the resulting graph | Different time windows produce different route evidence; switching back refreshes; bounded requests remain | +| Layout | Fit the Topology view into available screen space; Live Activity expands inside its allotted panel | Desktop and phone viewport containment, no page scrolling required to reach activity | +| Routes/Traces | Remove verbose explanations; compact trace rows while retaining tag/type/time/count/path/signal and keyboard inspection | Bilingual layout, narrow screens and existing drill-down behavior | +| Map live mode | Keep node icons visible by default, optionally dim them; reuse the existing observed-hop glow | Toggle, selection focus, live pulse and style changes; do not fabricate RX/TX events | +| Node detail | Add an Open full detail action and shareable full-page view, using existing node/report/neighbour data | Full-key identity, back/reload, sample bounds, loading/errors and graphs with real available samples | +| Observer cards | Sparklines for packet, battery, uptime and noise-floor series where supported | Reuse existing queries; preserve gaps/resets and match units/count definitions | +| Atlas telemetry | Show available telemetry for the exact node/observer identity | Never attach telemetry by display name or short prefix; absent data stays absent | + +## Telemetry investigation and feasible design + +An empty guest password is permission to authenticate to a repeater; it does not +make its directed response ciphertext public. MeshCore's repeater sends responses +with the requesting peer's ECDH shared secret. The Public/hashtag channel keys +apply to group messages, not that peer exchange. A passive MQTT observer cannot +derive the peer secret from two public keys or from the guest password. + +Sources: [MeshCore identity exchange](https://github.com/meshcore-dev/MeshCore/blob/main/src/Identity.h), +[repeater response/login code](https://github.com/meshcore-dev/MeshCore/blob/main/examples/simple_repeater/MyMesh.cpp), +[payload format](https://github.com/meshcore-dev/MeshCore/blob/main/docs/payloads.md). + +The [solar bot](https://github.com/n30nex/Canadaverse-MeshCore-Discord-Bot) already +uses an authenticated radio poller and stores decoded measurements. Its public +snapshot deliberately strips credentials and node keys, so it is not by itself an +identity-safe telemetry source for Beacon. + +Implementation ladder: + +1. Display the telemetry Beacon already receives in observer status, only when + the full public key maps that observer to the displayed node. Reuse existing + time-bucket queries and show volts/units, freshness and gaps. +2. Add a separately configured collector export from the solar bot/companion + after a concrete schema is agreed: full target key, poller identity, receive + time, request correlation, source and typed decoded measurements. Private keys + and passwords stay on the poller. Beacon does not start network-wide RF polls. +3. Automatically enroll and validate that ingest, deduplicate repeated samples, reject invalid + values, bound retention and preserve source attribution before storing node + telemetry. Decode LPP only after the transport is decoded/authenticated. +4. Public-channel telemetry can be parsed as unverified reported values; a sender + name is not a cryptographic node identity and must not overwrite trusted data. + +Battery percentage, panel watts, energy and runtime are not inferred from voltage +alone. Guest permissions and available sensors differ by firmware and node. Keep +that distinction in the data contract rather than filling absent metrics. + +## Deployment boundary + +The user approved fresh preview history while preserving the old database intact. +The existing `beacon_dev` database is never reset or relabelled. Its private dump +was restored successfully and verified on and off the Pi before cutover. New core +history uses `beacon_post21_preview_20261002`; community telemetry belongs to the +independent `beacon_collector_preview` database. The 1.4 handoff and official +production hosts remain outside this experiment. + +## Collector worldwide configuration and congestion policy + +The user authorized a standalone USB/BLE collector, initially tested with the Pi +RemoteTerm radio and YKF - 1W Hespeler by full key. The radio's firmware identifies +as Heltec Wireless Paper; the user confirmed this is the intended RemoteTerm unit. +The user later assigned this radio to the collector continuously. RemoteTerm stays +stopped and disabled until the user asks for its return; its original contacts +and route/configuration recovery journal are preserved privately on the Pi. + +The collector config chooses the Beacon instance URL and MQTT host/port/TLS, +credentials by environment variable, and topic namespace. Nothing is hardcoded to +Canadaverse. Each repeater gets a configurable **1–72 hour** interval. Attempt times +are persisted before RF, including failed polls, so restarts never trigger an early +retry. Status and sensor telemetry alternate across intervals when both are wanted; +there is only one data request per repeater interval (plus required guest login). + +Default cadence is six hours. Requests are staggered. Queue depth and measured +local RX+TX airtime over at least a minute gate transmission; busy/unknown state +causes deferral, not catch-up bursts. Deferral can make the actual gap longer than +the configured interval. Local schedules cannot coordinate different collectors or +Beacon instances; shared polling leases are a future server integration gate. + +Collector reports use an independent Ed25519 signing key. Enrollment is automatic through a Beacon challenge signed by both the companion +and collector. No operator authorization or per-client allowlist is required. +HTTPS is the default delivery, with optional MQTT. Radio private keys and repeater +passwords never leave the local device/host. This proves key control, not hardware +attestation or independent sensor truth. Preserve source attribution, validate +signatures, destination, timestamps, replay IDs and measurement bounds, and apply +per-IP API limits plus per-radio report limits. + +## Updated collector acceptance, 2 October evening + +The end result is an easy My Atlas repeater card with battery sparklines and +optional environmental telemetry. Setup lists contacts already on the user's USB +or BLE companion, allows per-repeater hidden password/admin entry, chooses any +Beacon instance, and starts polling. No manual client approval is part of the +flow. Direct HTTPS avoids broker credentials; MQTT remains optional. + +First discovery uses flood, then uses learned routes; failures wait for the next +1–72 hour interval. Explicit telemetry request permission bytes include external +sensors where access permits. Keep channel 2 and actual units. Cards use 30 days +of collector history, bounded to 500 recent reports, so 72-hour polling remains +useful. Telemetry-only cards can be pinned by full key before an advertisement. + +The first blank-guest Hespeler test returned no usable reading. The user then +selected the solar bot's six configured repeaters and authorized their Pi-local +admin-password aliases. Across bounded tests and normal operation, Hilltop, +Weaver, Royal City, Starkey and Royal Relay have returned correlated readings. +Reservoir has not returned a usable response. Hilltop supplied channel-2 +12.4 C, 64% humidity and 985 hPa, plus 3.99 V battery; later readings may differ. +Royal City has multiple real samples, allowing a measured battery sparkline. + +Temporary test exceptions were confined to a private harness; the distributed +collector retains normal cooldown and congestion rules. Its latest SDK handling +subscribes before a fast reply can arrive and checks the exact response tag before +using decoded telemetry. No admin configuration commands or invented readings are +part of the flow. + +## Private repository and independent intake service + +The user explicitly chose **private repository + separate service**. GitHub +`n30nex/Beacon-Telemetry-Collector` is private; the existing license is unchanged. +`gateway/` is a separate Go process with its own PostgreSQL role/schema and optional +MQTT listener. It has no Beacon core package or migration dependencies and refuses +to initialize over a core schema. The Python companion collector remains separate +from the intake process as well. + +Route `/api/v1/node-telemetry/*` to this gateway before the ordinary Beacon API. +Authentication is automatic dual-key enrollment, with no per-user operator approval. +Trusted proxy addresses are explicit; arbitrary forwarded headers do not affect +rate limits. Health checks include database access. The frontend treats telemetry +as an optional API so the new pages can be integrated without requiring the service. + +Accepted prototype samples were copied byte-for-byte into the dedicated database +before removing the experimental core table. The unpublished embedded-gateway +server commits are preserved locally and must never be pushed as public Beacon +history. The public server candidate descends from their clean parent instead. + +Release alongside a future Beacon version (possibly 3.0) is still planning. Before +broad release, finish multi-collector per-repeater polling coordination, operator +abuse/revocation controls, long-running delivery/reconnect tests and packaged USB/BLE +onboarding. Key possession is not hardware or sensor attestation, and private +source does not itself enforce RF behavior. From c665a88810f70caecae8f8466ec84e192f252661 Mon Sep 17 00:00:00 2001 From: n30nex Date: Fri, 2 Oct 2026 22:10:07 -0400 Subject: [PATCH 56/69] docs: distinguish deployed source and verify normal hourly telemetry --- docs/n30nex-test-preview.md | 2 ++ docs/post-140-roadmap.md | 2 ++ docs/post-20-integration.md | 2 ++ docs/post-20-ui-telemetry.md | 4 +++- 4 files changed, 9 insertions(+), 1 deletion(-) diff --git a/docs/n30nex-test-preview.md b/docs/n30nex-test-preview.md index cb5c6f3..5cc3a90 100644 --- a/docs/n30nex-test-preview.md +++ b/docs/n30nex-test-preview.md @@ -2,6 +2,8 @@ ## Current: 2.1 experimental preview — 3 October 2026 UTC +The web branch subsequently advanced to **f7637aa** for a single trailing-blank-line cleanup. The actual deployed and full-suite-tested web source remains **4b189dff**, correctly identified by its public source archive; this formatting-only branch difference has not been relabelled as a new deployment. + The user approved **private collector repository + separate intake service**, fresh preview history with the old database intact, and continued exclusive collector use of the RemoteTerm radio. These direct decisions supersede older cutover/restore diff --git a/docs/post-140-roadmap.md b/docs/post-140-roadmap.md index 6fc629a..b427d79 100644 --- a/docs/post-140-roadmap.md +++ b/docs/post-140-roadmap.md @@ -2,6 +2,8 @@ ## Current delivery — 3 October 2026 UTC +The web branch subsequently advanced to **f7637aa** for a single trailing-blank-line cleanup. The actual deployed and full-suite-tested web source remains **4b189dff**, correctly identified by its public source archive; this formatting-only branch difference has not been relabelled as a new deployment. + The [Pi preview](https://canadaverse.org/beacon-dev/) now runs experimental **2.1.0-n30nex.1**, server **fe4c4156** / web **4b189dff**, including the checked upstream **af20beb / 0924260**. The 2.0.1 patch milestone and 2.1.0 feature tags diff --git a/docs/post-20-integration.md b/docs/post-20-integration.md index 7f40229..f5d0a7c 100644 --- a/docs/post-20-integration.md +++ b/docs/post-20-integration.md @@ -2,6 +2,8 @@ ## Current: 2.1 experimental preview — 3 October 2026 UTC +The web branch subsequently advanced to **f7637aa** for a single trailing-blank-line cleanup. The actual deployed and full-suite-tested web source remains **4b189dff**, correctly identified by its public source archive; this formatting-only branch difference has not been relabelled as a new deployment. + The user approved **private collector repository + separate intake service**, fresh preview history with the old database intact, and continued exclusive collector use of the RemoteTerm radio. These direct decisions supersede older cutover/restore diff --git a/docs/post-20-ui-telemetry.md b/docs/post-20-ui-telemetry.md index b11d3ce..6a84aef 100644 --- a/docs/post-20-ui-telemetry.md +++ b/docs/post-20-ui-telemetry.md @@ -126,7 +126,9 @@ admin-password aliases. Across bounded tests and normal operation, Hilltop, Weaver, Royal City, Starkey and Royal Relay have returned correlated readings. Reservoir has not returned a usable response. Hilltop supplied channel-2 12.4 C, 64% humidity and 985 hPa, plus 3.99 V battery; later readings may differ. -Royal City has multiple real samples, allowing a measured battery sparkline. +Royal City has multiple real samples with a longer gap. Hilltop’s next normal +hourly poll arrived through the public service, and all five telemetry sparklines +were verified in the deployed Atlas card. Temporary test exceptions were confined to a private harness; the distributed collector retains normal cooldown and congestion rules. Its latest SDK handling From 1e947563acfa89af296cc6cab4f0ad9e32f565e0 Mon Sep 17 00:00:00 2001 From: n30nex Date: Sat, 3 Oct 2026 06:44:39 -0400 Subject: [PATCH 57/69] docs: record the post-2.0 preview audit and bound deployment logs --- ROADMAP.md | 52 +++++++------ docker-deployment-type1/docker-compose.yml | 11 +++ .../server/docker-compose.yml | 10 +++ .../web/docker-compose.yml | 8 ++ docs/n30nex-test-preview.md | 31 +++++++- docs/operations.md | 15 +++- docs/post-140-roadmap.md | 66 +++++++---------- docs/post-20-audit-20261003.md | 74 +++++++++++++++++++ docs/post-20-integration.md | 31 +++++++- 9 files changed, 229 insertions(+), 69 deletions(-) create mode 100644 docs/post-20-audit-20261003.md diff --git a/ROADMAP.md b/ROADMAP.md index b10a6fa..5e3cddf 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,29 +1,33 @@ # Beacon parity and analytics roadmap -## Active experimental work and proposed 2.0 baseline — 30 September - -The latest maintainer discussion ends at **2.0.0**, with synchronized server/web -major/minor versions and independent patch levels. Flattened migrations and a clean -start are proposed but have not landed in the checked upstream refs. The next step -is to validate that published baseline on a separate database before extending the -feature work. The discussion is not authorization to wipe the Pi or reuse an old -migration journal with a new baseline. See the [version and migration direction](docs/post-140-roadmap.md#maintainer-direction--30-september-2026). - -A follow-up proposes 24-hour raw retention on dev and 7 days in production, and -reports that dev has moved to rotational storage. No retention change is applied -to the Pi by this discussion. Chart windows need to follow the effective retention -of their deployment; summary-backed history remains distinct from raw records. - -The contributor approved proceeding toward and beyond CoreScope feature parity. -The sequence is **validate the upcoming 2.0 baseline → preserve saved-route correctness (#183) and experimental My Atlas (#97) -→ node/trace dashboards and investigation → search/saved views/channel analytics -→ topology/distance/hash tools → bounded replay/reach/timing → regional crossing evidence**. -Phases 1 and 2 are combined on the requested `n30nex-test` branches in server, web and docs. The Pi preview runs this experimental composition, with daily upstream dev/main compatibility checks and no review pings. The route fix and Atlas are validated; node/trace dashboards follow baseline validation. Each phase ends in focused reviewable PRs, exact Pi validation, -updated source/changelog and retained rollback; release numbers remain a maintainer decision. -The earlier 1.4.0 handoff remains recorded below as historical evidence. Atlas stays -experimental; its inclusion in a stable 2.0 release is not yet decided. - -[Phases, contracts and acceptance](docs/post-140-roadmap.md) · [Current experimental preview](docs/n30nex-test-preview.md). +## Current: released 2.0 baseline and experimental preview — 3 October + +The Pi preview includes released server `0015430` and web `7a9770e`, plus My Atlas, +Topology, full node pages and exact route evidence. Its deployed revisions are +server **8c7fbb8** (`2.1.0-n30nex.2`) and web **7e139d5** (`2.1.1-n30nex.1`). +Both contain the checked upstream main/dev branches. The separate private Collector +runs alpha.4; its implementation and prototype history are absent from public Beacon. + +The new 001 database baseline is active; the old database and verified recovery +copies remain intact. At the user's request, preview retention matches the released +defaults: **7 days raw, 90 days summaries, 31 days observer telemetry and 14 days +routes**. This replaces older preview-policy notes below. No production service, +configuration or database was changed. + +753 server tests and 1,259 frontend tests passed, plus native PostgreSQL/build/lint, +31 preview API checks and exact public assets/source checks. Topology camera +continuity, node/telemetry error handling and missing-number validation are fixed. +Deployment-log rotation is prepared in these templates under docs issue #13. + +Next: complete node/fleet history and graphical trace analysis, then shared search, +saved filters, channel analytics, prefix/distance tools, retained replay and area +filtering. Collector wider-alpha gates remain separate. Experimental PRs stay +draft with no review pings; daily automation remains paused. Older checkpoints below +are historical, including their version, retention and recovery instructions. + +[Audit and remaining gates](docs/post-20-audit-20261003.md) · +[Parity roadmap](docs/post-140-roadmap.md) · +[Current experimental preview](docs/n30nex-test-preview.md). ## Recorded 1.4.0 checkpoint — superseded on the Pi by the experiment diff --git a/docker-deployment-type1/docker-compose.yml b/docker-deployment-type1/docker-compose.yml index 8676117..015ca97 100644 --- a/docker-deployment-type1/docker-compose.yml +++ b/docker-deployment-type1/docker-compose.yml @@ -1,3 +1,9 @@ +x-logging: &bounded-logging + driver: json-file + options: + max-size: "10m" + max-file: "3" + services: app: image: ${BEACON_SERVER_IMAGE:?Set BEACON_SERVER_IMAGE to the reviewed server tag or digest} @@ -12,6 +18,7 @@ services: redis: condition: service_healthy restart: unless-stopped + logging: *bounded-logging db: image: postgres:16-alpine @@ -29,6 +36,7 @@ services: timeout: 5s retries: 5 restart: unless-stopped + logging: *bounded-logging redis: image: redis:7-alpine @@ -42,6 +50,7 @@ services: timeout: 5s retries: 5 restart: unless-stopped + logging: *bounded-logging caddy: image: caddy:2-alpine @@ -61,6 +70,7 @@ services: web: condition: service_started restart: unless-stopped + logging: *bounded-logging web: image: ${BEACON_WEB_IMAGE:?Set BEACON_WEB_IMAGE to the reviewed web tag or digest} @@ -76,6 +86,7 @@ services: app: condition: service_started restart: unless-stopped + logging: *bounded-logging # Fixed subnet so data/app/config.yaml can trust Caddy in server.trusted_proxies. # Pick another private /24 if this one is already in use on the host. diff --git a/docker-deployment-type2/server/docker-compose.yml b/docker-deployment-type2/server/docker-compose.yml index 2a69d51..533d953 100644 --- a/docker-deployment-type2/server/docker-compose.yml +++ b/docker-deployment-type2/server/docker-compose.yml @@ -1,3 +1,9 @@ +x-logging: &bounded-logging + driver: json-file + options: + max-size: "10m" + max-file: "3" + services: app: image: ghcr.io/meshcore-beacon/beacon-server:latest @@ -12,6 +18,7 @@ services: redis: condition: service_healthy restart: unless-stopped + logging: *bounded-logging db: image: postgres:16-alpine @@ -29,6 +36,7 @@ services: timeout: 5s retries: 5 restart: unless-stopped + logging: *bounded-logging redis: image: redis:7-alpine @@ -42,6 +50,7 @@ services: timeout: 5s retries: 5 restart: unless-stopped + logging: *bounded-logging caddy: image: caddy:2-alpine @@ -62,6 +71,7 @@ services: app: condition: service_started restart: unless-stopped + logging: *bounded-logging # Fixed subnet so data/app/config.yaml can trust Caddy in server.trusted_proxies. # Pick another private /24 if this one is already in use on the host. diff --git a/docker-deployment-type2/web/docker-compose.yml b/docker-deployment-type2/web/docker-compose.yml index bf9a013..9a710a6 100644 --- a/docker-deployment-type2/web/docker-compose.yml +++ b/docker-deployment-type2/web/docker-compose.yml @@ -1,3 +1,9 @@ +x-logging: &bounded-logging + driver: json-file + options: + max-size: "10m" + max-file: "3" + services: caddy: image: caddy:2-alpine @@ -11,6 +17,7 @@ services: - ./data/Caddy/data:/data/caddy/ - ./data/Caddy/config:/config/caddy/ restart: unless-stopped + logging: *bounded-logging beacon-web: image: ${BEACON_WEB_IMAGE:?Set BEACON_WEB_IMAGE to a reviewed release tag or digest} @@ -29,3 +36,4 @@ services: # Notice above the header, e.g. on a dev instance; unset = none - VITE_BANNER=${VITE_BANNER:-} restart: unless-stopped + logging: *bounded-logging diff --git a/docs/n30nex-test-preview.md b/docs/n30nex-test-preview.md index 5cc3a90..92acf7e 100644 --- a/docs/n30nex-test-preview.md +++ b/docs/n30nex-test-preview.md @@ -1,6 +1,35 @@ # Experimental n30nex-test preview -## Current: 2.1 experimental preview — 3 October 2026 UTC +## Current: released 2.0 base and verified preview — 3 October 2026 + +The [Pi preview](https://canadaverse.org/beacon-dev/) runs server **8c7fbb8** and +web **7e139d5**, retaining My Atlas, Topology and node/route work on top of the +published **2.0.0** release. Experimental tags are `v2.1.0-n30nex.2` (server) and +`v2.1.1-n30nex.1` (web). Stable release and production ownership remain upstream. + +At the user's request, the preview matches released defaults: **7-day raw, +90-day summaries, 31-day observer telemetry, 14-day routes**. The runtime values +were verified. The previous 30-day-summary documentation was wrong; the previous +runtime already used 90 days. Legacy history is still intact. + +This patch fixes camera resets on refreshed Topology data/layout, retry/error +feedback on node and telemetry pages, and optional telemetry collections. Private +Collector alpha.4 rejects missing/null readings instead of inventing zero; its +source and intake database remain separate from Beacon core. The current signed +HTTPS path passed real companion checks with no extra RF polling or fake samples. + +Validation: **753 server tests, 1,259 web tests**, native PostgreSQL/build/vet/lint, +**31 preview API checks**, exact public assets/source archives and sampled browser +journeys. Two optional backup tests and physical Safari/BLE coverage are not claimed. +All changes are preview/repository work; the earlier production inspection was +read-only and stopped when the user redirected the scope. + +[Audit findings and next gates](post-20-audit-20261003.md) · +[Merge sequence](post-140-roadmap.md) · +[Current preview source](https://canadaverse.org/beacon-dev/source.html). + + +## Historical: early 2.1 experimental preview — 3 October 2026 UTC The web branch subsequently advanced to **f7637aa** for a single trailing-blank-line cleanup. The actual deployed and full-suite-tested web source remains **4b189dff**, correctly identified by its public source archive; this formatting-only branch difference has not been relabelled as a new deployment. diff --git a/docs/operations.md b/docs/operations.md index ad755ea..a1e91f9 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -19,16 +19,23 @@ Beacon has no health endpoint yet. Use these instead: ## Logs -Logs go to stderr and Docker collects them. Docker's default `json-file` driver does not -rotate, so on a long-running host set a cap, either in `/etc/docker/daemon.json` for every -container or per service in `docker-compose.yml`: +Logs go to stderr and Docker collects them. The supplied all-in-one and split deployment +templates cap each service's Docker logs at three files of about 10 MB each. Their shared +`bounded-logging` block applies to every service. Adjust that policy for your retention needs: ```yaml logging: driver: json-file - options: { max-size: "50m", max-file: "5" } + options: { max-size: "10m", max-file: "3" } ``` +Docker's default `json-file` driver is unbounded when no rotation options are configured. +Existing containers do not adopt a changed logging policy on restart: recreate the affected +services during a planned maintenance window, preserving their data volumes. Inspect the +effective policy with `docker inspect --format '{{json .HostConfig.LogConfig}}' `. +Do not remove database volumes to apply a logging change. This policy is separate from +Caddy's rotated access-log files. + Set `log.level` (`debug`, `info`, `warn`, `error`; default `info`) and `log.format` (`text` or `json`) in `config.yaml`, or override them with `LOG_LEVEL` and `LOG_FORMAT`. Invalid values stop startup. diff --git a/docs/post-140-roadmap.md b/docs/post-140-roadmap.md index b427d79..f4a981e 100644 --- a/docs/post-140-roadmap.md +++ b/docs/post-140-roadmap.md @@ -1,44 +1,32 @@ # Beacon roadmap: 2.0 integration and CoreScope parity -## Current delivery — 3 October 2026 UTC - -The web branch subsequently advanced to **f7637aa** for a single trailing-blank-line cleanup. The actual deployed and full-suite-tested web source remains **4b189dff**, correctly identified by its public source archive; this formatting-only branch difference has not been relabelled as a new deployment. - -The [Pi preview](https://canadaverse.org/beacon-dev/) now runs experimental -**2.1.0-n30nex.1**, server **fe4c4156** / web **4b189dff**, including the checked -upstream **af20beb / 0924260**. The 2.0.1 patch milestone and 2.1.0 feature tags -are published on contribution forks; they are not upstream stable releases. - -Delivered in the preview: - -- Topology left-pan/right-orbit/wheel zoom, scene-anchored labels, route-window - refresh, viewport-sized Live Activity, region isolation and fullscreen. -- Full node pages with bounded report summaries, observer sparklines, visible live - map icons with reception glow, and compact Routes/Traces. -- My Atlas battery and environmental telemetry via the **separate Beacon Collector - service/database**. The collector repository is private; its license is unchanged. - Automatic enrollment does not require operator approval. -- Fresh 2.0 baseline history, preserving the old database intact and its verified - on/off-Pi recovery dump. 72-hour raw/30-day summary retention remains. - -Verified: **753 server tests, 1,255 web tests**, native PostgreSQL/build/lint/vet, -3,200 replay inputs with zero fixture drops, 30 public assets and both corresponding -source archives, signed public HTTPS delivery, duplicate rejection and invalid -signature rejection. Initial 20-second replay timing failed under concurrent load; -the retained 60-second-budget run passed in 22.80 seconds. Two optional backup tests -remain skipped. Browser checks cover live traffic, camera controls, route refresh, -region isolation, fullscreen, full-node details and French phone containment. - -Five of the six selected solar repeaters have returned readings. Reservoir has not. -Hilltop supplied channel-2 temperature, humidity and pressure plus battery. Graphs -require later successful readings and retain gaps; no values are fabricated. The -leased radio stays on the normal hourly collector schedule and RemoteTerm remains -suspended at the user's request. - -Rollback is `evidence/post21-final-20261003/deploy-preview.py rollback`, checkpoint -`post21-cutover-20261003T014506Z`. The core and telemetry databases remain separate. -Review requests and the daily compatibility automation remain on hold. Official -production deployment and stable version selection belong to the maintainers. +## Current: released 2.0 base and verified preview — 3 October 2026 + +The [Pi preview](https://canadaverse.org/beacon-dev/) runs server **8c7fbb8** and +web **7e139d5**, retaining My Atlas, Topology and node/route work on top of the +published **2.0.0** release. Experimental tags are `v2.1.0-n30nex.2` (server) and +`v2.1.1-n30nex.1` (web). Stable release and production ownership remain upstream. + +At the user's request, the preview matches released defaults: **7-day raw, +90-day summaries, 31-day observer telemetry, 14-day routes**. The runtime values +were verified. The previous 30-day-summary documentation was wrong; the previous +runtime already used 90 days. Legacy history is still intact. + +This patch fixes camera resets on refreshed Topology data/layout, retry/error +feedback on node and telemetry pages, and optional telemetry collections. Private +Collector alpha.4 rejects missing/null readings instead of inventing zero; its +source and intake database remain separate from Beacon core. The current signed +HTTPS path passed real companion checks with no extra RF polling or fake samples. + +Validation: **753 server tests, 1,259 web tests**, native PostgreSQL/build/vet/lint, +**31 preview API checks**, exact public assets/source archives and sampled browser +journeys. Two optional backup tests and physical Safari/BLE coverage are not claimed. +All changes are preview/repository work; the earlier production inspection was +read-only and stopped when the user redirected the scope. + +[Audit findings and next gates](post-20-audit-20261003.md) · +[Merge sequence](post-20-integration.md) · +[Current preview source](https://canadaverse.org/beacon-dev/source.html). ## Remaining parity and next phases diff --git a/docs/post-20-audit-20261003.md b/docs/post-20-audit-20261003.md new file mode 100644 index 0000000..c5c3a6d --- /dev/null +++ b/docs/post-20-audit-20261003.md @@ -0,0 +1,74 @@ +# Post-2.0 audit: preview delivery and remaining gates + +3 October 2026. Production was inspected read-only before the user narrowed the +scope to preview only. No live service, configuration or database was modified. + +The released baseline is server `0015430` and web `7a9770e` (v2.0.0). The preview +now runs server `8c7fbb8`, web `7e139d5`, and the separate private Collector +`aa7ddf0` / alpha.4. Web main was source-identical to dev `6570c95`; its squashed +history was reconciled without dropping the experimental features or force-pushing. + +## Fixed in the preview + +- Preserve Topology pan/angle/zoom/region focus across refreshed graph data and + layout changes. Explicit Fit remains available. +- Distinguish node-load failures from missing nodes, offer retry, and refresh + detail/neighbour data at bounded intervals. +- Show an initial Collector read failure and handle status-only/sensor-only + optional collections without crashing a card. +- Reject signed reports with missing/null numbers; preserve genuine zero readings. + The private Collector issue #1 and signed-envelope regression record the fix. +- Match the user's requested upstream retention: raw 7 days, summaries 90 days, + observer telemetry 31 days and routes 14 days. The prior documentation incorrectly + claimed 30-day summaries; the runtime was already using the 90-day default. + +## Deployment-template hardening + +[Docs issue #13](https://github.com/MeshCore-Beacon/beacon-docs/issues/13) tracks +unbounded Docker stdout/stderr logs. A 10 MB × 3 policy is prepared for all 11 +services in the all-in-one and split templates. Existing containers require planned +recreation to adopt it; data volumes must be retained. Caddy access-log rotation is +separate. These template changes were not applied to production. + +## Evidence and limits + +753 server tests and 1,259 web tests (152 files) passed, with native PostgreSQL, +build/vet/lint and 31 post-deployment preview API checks. Two opt-in backup/export +tests were skipped. Public hashes match 30 frontend files and both corresponding +source archives. Exact Actions-built alpha.4 artifacts passed Windows/Linux and +PostgreSQL checks. Real companion enrollment/delivery, replay deduplication, +invalid-signature rejection and missing-number rejection passed on preview. + +Browser checks exercise the new pages and the existing main navigation. Physical +Safari and all BLE/Windows device combinations remain untested. The historical +3,200-input queue replay is not relabelled as a new benchmark. Small samples from +a young deployment do not establish long-term throughput, losslessness or capacity. + +The production sample showed healthy containers/database activity and many calls +to retired CoreScope-style API paths. Old tabs/integrations are a hypothesis, not +an established caller identity. Refresh old tabs/update integrations without +clearing saved browser data. WebSocket idle warnings need a targeted reproduction +before heartbeat policy changes; no new-client failure was established. + +## Next phases + +1. Finish complete node/fleet history beyond the current 200-report sample and + graphical trace chronology/reach. +2. Global search, saved mesh/Atlas filters and channel analytics. +3. Prefix/collision tools, distance and repeated-route analysis. +4. Retained packet playback and geographic-area filtering. +5. Closed-alpha Collector cohort: private test hub prepared, with test records and + package instructions. Choose testers/distribution later; no invitations were + sent. Finish per-repeater coordination across clients, abuse/revocation controls, + physical USB/BLE coverage and longer reconnect/outbox qualification first. + +Five configured solar repeaters have returned data; Reservoir has not. Missing +readings remain gaps. Source signing proves key possession, not hardware/sensor +attestation. The radio remains leased to the collector, with RemoteTerm suspended. +Normal 1–72-hour cadence and congestion checks remain in the distributed package. + +The active/legacy databases and verified recovery copies remain intact. Current +paired recovery is the Pi `evidence/post20-audit-20261003/deploy-audit.py rollback`, +checkpoint `post20-audit-20261003T094351Z`, retaining the user's upstream retention +policy. The private Collector implementation is not in Beacon server's history. +Experimental PRs stay draft and the daily automation remains paused. diff --git a/docs/post-20-integration.md b/docs/post-20-integration.md index f5d0a7c..b805c08 100644 --- a/docs/post-20-integration.md +++ b/docs/post-20-integration.md @@ -1,6 +1,35 @@ # My Atlas and Topology after Beacon 2.0 -## Current: 2.1 experimental preview — 3 October 2026 UTC +## Current: released 2.0 base and verified preview — 3 October 2026 + +The [Pi preview](https://canadaverse.org/beacon-dev/) runs server **8c7fbb8** and +web **7e139d5**, retaining My Atlas, Topology and node/route work on top of the +published **2.0.0** release. Experimental tags are `v2.1.0-n30nex.2` (server) and +`v2.1.1-n30nex.1` (web). Stable release and production ownership remain upstream. + +At the user's request, the preview matches released defaults: **7-day raw, +90-day summaries, 31-day observer telemetry, 14-day routes**. The runtime values +were verified. The previous 30-day-summary documentation was wrong; the previous +runtime already used 90 days. Legacy history is still intact. + +This patch fixes camera resets on refreshed Topology data/layout, retry/error +feedback on node and telemetry pages, and optional telemetry collections. Private +Collector alpha.4 rejects missing/null readings instead of inventing zero; its +source and intake database remain separate from Beacon core. The current signed +HTTPS path passed real companion checks with no extra RF polling or fake samples. + +Validation: **753 server tests, 1,259 web tests**, native PostgreSQL/build/vet/lint, +**31 preview API checks**, exact public assets/source archives and sampled browser +journeys. Two optional backup tests and physical Safari/BLE coverage are not claimed. +All changes are preview/repository work; the earlier production inspection was +read-only and stopped when the user redirected the scope. + +[Audit findings and next gates](post-20-audit-20261003.md) · +[Merge sequence](post-140-roadmap.md) · +[Current preview source](https://canadaverse.org/beacon-dev/source.html). + + +## Historical: early 2.1 experimental preview — 3 October 2026 UTC The web branch subsequently advanced to **f7637aa** for a single trailing-blank-line cleanup. The actual deployed and full-suite-tested web source remains **4b189dff**, correctly identified by its public source archive; this formatting-only branch difference has not been relabelled as a new deployment. From 14aae17d3be5b73f7adab8f7d41fd09a047fd49d Mon Sep 17 00:00:00 2001 From: n30nex Date: Sat, 3 Oct 2026 11:21:24 -0400 Subject: [PATCH 58/69] docs: scope the proposed 2.1 Atlas and Collector work --- ROADMAP.md | 9 +- docs/beacon-21-collector-design.md | 210 +++++++++++++++++++++++++++++ docs/post-140-roadmap.md | 11 +- docs/post-20-ui-telemetry.md | 7 + 4 files changed, 233 insertions(+), 4 deletions(-) create mode 100644 docs/beacon-21-collector-design.md diff --git a/ROADMAP.md b/ROADMAP.md index 5e3cddf..5664f09 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -19,9 +19,12 @@ configuration or database was changed. continuity, node/telemetry error handling and missing-number validation are fixed. Deployment-log rotation is prepared in these templates under docs issue #13. -Next: complete node/fleet history and graphical trace analysis, then shared search, -saved filters, channel analytics, prefix/distance tools, retained replay and area -filtering. Collector wider-alpha gates remain separate. Experimental PRs stay +The latest discussion proposes **Atlas and Collector as the 2.1 focus**. The +[planning scratchpad](docs/beacon-21-collector-design.md) compares MeshCore HA's +traffic controls and records USB/TCP, per-radio budgets, shared repeater leases +and MQTT decisions. Topology remains in the preview; full node/fleet history, +graphical trace analysis, shared search, channel analytics and replay remain on the +broader parity roadmap. Collector wider-alpha gates remain explicit. Experimental PRs stay draft with no review pings; daily automation remains paused. Older checkpoints below are historical, including their version, retention and recovery instructions. diff --git a/docs/beacon-21-collector-design.md b/docs/beacon-21-collector-design.md new file mode 100644 index 0000000..ecb7392 --- /dev/null +++ b/docs/beacon-21-collector-design.md @@ -0,0 +1,210 @@ +# Beacon 2.1: Atlas and Collector planning scratchpad + +3 October 2026. This is a source-backed planning note, not a deployment instruction +or an accepted upstream release contract. The supplied discussion proposes Atlas +and Collector as the 2.1 focus, USB/Wi-Fi first, and MQTT forwarding. Maintainer +thoughts and the future private repository handoff are still pending. No access, +repository transfer, MQTT switch, RF policy change or remote-control permission is +inferred from that discussion. + +## Scope and sequence + +| Milestone | Proposed scope | Status | +|---|---|---| +| **2.1** | My Atlas with useful battery/environment graphs; a separate Collector with simple contact selection and conservative polling | Proposed release focus. USB preview exists; wider rollout still has gates below. | +| **2.1.x** | Compatibility, correctness and usability fixes for the accepted 2.1 scope | Patch releases; no automatic jump to a new major. | +| Later transport work | BLE qualification and browser collectors | BLE code exists but is not fully hardware-qualified. It need not block the USB/TCP first release. | +| **2.5, provisional** | Authenticated observer/repeater console or MQTT control | Exploration only. Read-only telemetry enrollment must not grant administration. | +| Future firmware | Reuse the agreed contract in observer firmware | An interoperability goal, not a prerequisite or a firmware-change request. | + +Topology and the other experimental pages remain available in the Pi preview. +Their later integration and the broader CoreScope parity backlog are separate +from the proposed Atlas/Collector release cut. The previous possible 3.0 pairing +was exploratory; this discussion proposes a 2.1 path, conditional on its gates. +Work remains on `n30nex-test`, with existing PRs in draft and no review pings. + +Keep requirements specific to Beacon. A MeshMapper change belongs in the plan +only when a concrete Beacon use case and a minimal API requirement are identified. +This document is the shared scratchpad; no new bot, account or task is required. + +## Existing decisions to preserve + +- Any compatible Beacon instance can receive data; Canadaverse remains the initial + preview destination. Users select contacts by full public key and can provide a + guest/password/admin credential locally. The operator does not approve each client. +- User-selected intervals remain **1–72 hours**, with six hours the normal default. + A timeout, restart, reconnect or manual refresh must not bypass the minimum. +- Initial discovery floods; successful learned routes are reused. Channel-2 LPP + readings retain their channel, type and units. Missing readings stay missing. +- Intake is a separate service/database. Collector source remains private under + its existing license. Repository privacy is not an RF enforcement mechanism. +- The user-facing result is a pinned Atlas card with real readings, history, + freshness and gaps. Full-key identity, collector attribution and optional API + compatibility remain necessary for independent Atlas integration. + +## What MeshCore HA actually does + +Inspected public source, pinned rather than relying on moving release notes: + +- Stable **v2.10.0**, `0f99da64be8a0ab6eacd4e87246f2a70e624f2b6`. +- Beta **v3.0.0-beta**, `e9e60552f671e6c6d765c4826c18839ac54f22ce`. + The beta tag is updated in place; this SHA identifies the reviewed snapshot. + +The stable implementation starts with a **20-credit** shared bucket and refills +one credit per **120 seconds**: about **30 requests/hour**, with an initial burst +allowance. Login and data requests consume credits separately. A depleted bucket +skips work; some stable polling paths count that skip as a failure. Credits are +held in memory, so this is not a persistent, network-wide polling lease. +Sources: [bucket](https://github.com/meshcore-dev/meshcore-ha/blob/0f99da64be8a0ab6eacd4e87246f2a70e624f2b6/custom_components/meshcore/rate_limiter.py), +[constants](https://github.com/meshcore-dev/meshcore-ha/blob/0f99da64be8a0ab6eacd4e87246f2a70e624f2b6/custom_components/meshcore/const.py#L206), +[polling callers](https://github.com/meshcore-dev/meshcore-ha/blob/0f99da64be8a0ab6eacd4e87246f2a70e624f2b6/custom_components/meshcore/coordinator.py#L1295). + +The inspected beta's governed policy separates a flat per-radio budget into: + +| Lane | Burst credits | Refill credits/hour | +|---|---:|---:| +| Flood | 5 | 20 | +| Direct | 20 | 120 | +| User messages | 10 | 60 | + +These are HA's numbers, **not adopted Beacon defaults or measured airtime limits**. +Adding tracked nodes shares that budget instead of growing it. Governed denial +means deferral rather than a node failure. It persists lane credits and node +schedules, exposes the next eligible time, and applies longer jittered backoff to +flood failures. New beta installations are assigned governed mode; an existing +entry without the setting still resolves to legacy. The module's opening comment +and earlier beta release text are less precise than those current constants. +Sources: [policy and costs](https://github.com/meshcore-dev/meshcore-ha/blob/e9e60552f671e6c6d765c4826c18839ac54f22ce/custom_components/meshcore/traffic.py#L59), +[new/existing defaults](https://github.com/meshcore-dev/meshcore-ha/blob/e9e60552f671e6c6d765c4826c18839ac54f22ce/custom_components/meshcore/const.py#L211), +[persistence](https://github.com/meshcore-dev/meshcore-ha/blob/e9e60552f671e6c6d765c4826c18839ac54f22ce/custom_components/meshcore/coordinator.py#L883), +[policy tests](https://github.com/meshcore-dev/meshcore-ha/blob/e9e60552f671e6c6d765c4826c18839ac54f22ce/tests/test_traffic.py#L233). + +Beacon should use the budgeting/deferral ideas while retaining its own stronger +per-target minimum. HA's routed retry spacing and immediate path-healing attempts +are unsuitable defaults for an hourly telemetry collector. Its debounced state +save also does not replace Beacon's requirement to persist the attempt before RF. + +**Traffic credits and authentication tokens solve different problems.** HA's +inspected MQTT uploader creates a cached, expiring, optionally audience-bound JWT +using a private key exported from the connected companion. That authenticates a +broker connection; it is not permission to transmit on RF or proof of sensor truth. +Beacon's current challenge signing can keep the radio private key on the device. +An MQTT credential design should retain that property instead of copying HA's +private-key export mechanism. +Source: [beta MQTT signing and export path](https://github.com/meshcore-dev/meshcore-ha/blob/e9e60552f671e6c6d765c4826c18839ac54f22ce/custom_components/meshcore/mqtt_uploader.py#L592). + +This was source inspection, not hardware qualification or a security audit of HA. +HA is MIT licensed; any future code reuse must retain the required notices. This +planning change copies no HA implementation into Beacon. + +## Proposed Collector design + +### Radio traffic + +Keep one serialized polling loop. Gate it with the existing persisted target +schedule and local queue/airtime checks, then add a shared **per-radio** budget +with separate flood and direct credits. Both login and data requests count; local +USB statistics reads and resending an already-collected HTTPS/MQTT report do not +create new RF polls. Persist debits before transmission, preserve them through +restarts, and avoid a full startup/catch-up burst. A second configuration must not +open the same radio and obtain a second independent budget. + +Budget exhaustion, busy airtime and unavailable airtime are **deferred** states, +with a reason and next eligible time. They do not label a repeater offline. Real +timeouts or rejected credentials remain distinct. Repeated failures get bounded +backoff and an explicit paused/retry state; the chosen nominal interval is not a +promise that congestion can never delay a sample. + +For a wider cohort, obtain an atomic **per-repeater lease before RF** from the +Collector service. Scope it to full radio/collector/repeater keys and the service +instance; preserve the cooldown even if the client crashes or the poll fails. +Two clients must not both poll the same repeater at once. A server ingest quota +only rejects a report after transmission and cannot provide this guarantee. +Instance-local leases also do not coordinate separate Beacon operators: any +cross-instance authority or federation must be an explicit later decision. + +Hop policy remains open. A maximum permitted **known route length** can defer +long direct paths. It does not cap how far a flood propagates, and scope labels +are not automatically RF hop limits. Confirm firmware support and behavior before +promising a hard flood cap; do not silently change a user's radio configuration. +Use conservative flood budgets and real airtime observations in the meantime. + +Exact credit rates, burst sizes, retry backoff and any hop threshold need agreement +and measured preview results. HA's constants are a reference, not proof they suit +this mesh, RF profile or a worldwide rollout. + +### Connections and delivery + +Qualify **USB and TCP to a Wi-Fi companion first**. TCP is the network transport +to the radio; it is distinct from MQTT delivery to the server. The MeshCore Python +SDK documents serial, TCP and BLE factories, but our current Collector exposes +only serial/BLE. TCP setup, validation, reconnect and physical testing are work +still to do. Treat a raw companion TCP endpoint as a trusted-LAN connection until +its authentication/encryption is verified; do not expose it publicly. +Source: [official SDK connection documentation](https://github.com/meshcore-dev/meshcore_py/blob/main/README.md#connecting-to-your-device). + +Keep the existing signed report schema and automatic enrollment. MQTT forwarding +already has an optional adapter and the separate `beacon/telemetry/v1/` +namespace. A proposed MQTT-first release needs automatic, expiring broker +credentials restricted to that client's publish topic, a broker-side verification +contract, revocation and an intake acceptance receipt. A broker PUBACK alone is +not proof that Beacon validated/stored a report. Reuse the saved reading for +delivery retries and deduplicate it across transports. + +The current preview continues to use HTTPS. No broker configuration or delivery +default changes were made for this discussion. HTTPS remains useful for enrollment +and a fallback; selecting MQTT as the release default is an open decision. + +Remote command/control is a separate future permission surface. Telemetry upload +credentials must not allow subscriptions or publications that execute commands. +Before any control feature: explicit device-owner opt-in, narrowly allowed +commands, signed short-lived requests, replay protection, revocation, audit records +and a local disable switch. No remote shell, password upload or administration +channel is part of the proposed 2.1 telemetry scope. + +### Atlas and operator visibility + +Keep setup focused: connect a companion, select saved repeaters, choose access and +interval, and pin the resulting full-key cards. Show battery and available channel-2 +environment graphs with units, source and last successful sample. A single sample +is a reading; history requires subsequent successes. Do not infer battery percentage +or manufacture points for missed polls. + +Add concise collector health: connected/disconnected, waiting for its interval, +deferred for airtime/budget/another collector, login denied, timed out, delivered, +or waiting for server acceptance. Show the next eligible poll and last delivery +without exposing passwords, radio private keys or enrollment tokens. Atlas must +remain useful when the optional Collector service is absent. + +## Work packages and acceptance + +| Order | Focused package | Completion evidence | +|---|---|---| +| 1 | Atlas integration and setup/health contract | Standalone Atlas against released Beacon; no required collector core migration; sample/gap/error states in English/French. | +| 2 | Per-radio flood/direct budget and explicit deferrals | Every RF path charged; exhausted/busy states send nothing; crashes, clock changes and restarts cannot refill early; existing 1–72-hour target contract passes. | +| 3 | Shared repeater leases | Concurrent clients: only one grant; expiry/crash/failure retain cooldown; fresh keys/re-enrollment cannot bypass radio/target quotas; service outage does not create a fleet of uncoordinated polls. | +| 4 | TCP companion and USB packaging | Exact supported firmware, saved-contact selection, identity change, disconnect/reconnect and exclusive radio ownership; physical Windows/Linux USB and LAN TCP results. | +| 5 | MQTT ingestion/acceptance and automatic credentials | Wrong topic/audience/key and expired/replayed messages rejected; reconnect/outbox delivery deduplicated; denial/revocation visible; no extra RF to retry upload. | +| 6 | Small closed alpha | Named cohort, agreed defaults, measured airtime and sample success, pause/recovery instructions; expand only after the earlier gates. | + +These are proposed packages, not claims of completed implementation. Keep changes +small and independently reviewable. No new feature package was deployed while +writing this note, and the wider parity backlog is retained. + +## Decision log for the next discussion + +| Question | Suggested starting point | Still needed | +|---|---|---| +| What ships as 2.1? | Atlas + a bounded Collector alpha | Agreement on acceptance and whether wider distribution is gated to a later patch/feature release. | +| Which transport? | USB and LAN TCP qualification first | Supported Wi-Fi firmware/device; BLE/browser later coverage. | +| MQTT default? | Reuse one signed payload; HTTPS enrollment; automatic scoped broker credentials | Broker/verifier owner, topic ACL and intake-receipt contract. | +| How much RF? | Preserve 1–72h target intervals; add independent flood/direct caps | Agreed measured budgets and failure policy; no copied HA defaults. | +| How far? | Optional known-route policy, conservative flood handling | Supported firmware controls and measured coverage; no claimed hard cap yet. | +| Multiple collectors? | One instance grants target leases before polling | Explicit handling of separate Beacon instances. | +| Repository handoff? | Keep current private repository/license until the destination is verified | Actual invitation/permissions and chosen migration path. | + +Existing limits: signing proves key possession, not hardware attestation; a +cooperative client policy does not stop unrelated or modified radios transmitting. +As of the latest preview check, five selected repeaters have real readings. +Reservoir's target was corrected from `bbeb6123` to SolarWatch's `9292f89e` identity; +the first corrected poll timed out and is not counted as a successful sample. diff --git a/docs/post-140-roadmap.md b/docs/post-140-roadmap.md index f4a981e..4b6ec27 100644 --- a/docs/post-140-roadmap.md +++ b/docs/post-140-roadmap.md @@ -30,6 +30,15 @@ read-only and stopped when the user redirected the scope. ## Remaining parity and next phases +The 3 October discussion proposes **2.1 focused on My Atlas and Collector**, with +USB/TCP companion qualification first. See the [2.1 planning scratchpad and +MeshCore HA comparison](beacon-21-collector-design.md) for existing safeguards, +the proposed flood/direct budgets, shared repeater leases and MQTT acceptance +requirements. MQTT control/console work is provisionally later (2.5); BLE/browser +qualification and observer-firmware integration are later work. These are planning +proposals, not changes to the running preview or permission to enable control. +Topology stays available in the experiment; the broader parity backlog follows. + | Area | Remaining work | |---|---| | Node/repeater analytics | Full history beyond the bounded 200-report sample; signal/hop distributions, heatmaps, hearing coverage and comparable fleet metrics. | @@ -37,7 +46,7 @@ read-only and stopped when the user redirected the scope. | Finding and channel analysis | Global search, saved mesh/Atlas filters, channel activity and sender/hearing comparisons. | | Network tools | Prefix collisions/checker, geographic distance analysis, repeated subpaths and route alternatives. | | Replay and geography | Retained packet playback/seek/speed, reach/timing analysis and GPS-area filtering. | -| Collector release readiness | Cross-client per-repeater leases, abuse/revocation controls, packaged USB/BLE onboarding, long-run reconnect/delivery tests and diagnosis of unresponsive targets. A possible Beacon 3.0 pairing remains planning. | +| Collector release readiness | Proposed 2.1 Atlas/Collector focus: per-radio flood/direct budgets, cross-client repeater leases, USB/TCP qualification, automatic MQTT credentials and acceptance receipts, abuse/revocation controls and long-run reconnect tests. BLE/browser and remote control follow later. | | Smaller optional work | QR sharing, theme/layout import/export and operator diagnostics. | ## Comparison baseline and scope diff --git a/docs/post-20-ui-telemetry.md b/docs/post-20-ui-telemetry.md index 6a84aef..27acb71 100644 --- a/docs/post-20-ui-telemetry.md +++ b/docs/post-20-ui-telemetry.md @@ -1,5 +1,12 @@ # Post-2.0 UI and node telemetry batch +**Current planning update, 3 October:** see [Beacon 2.1: Atlas and Collector](beacon-21-collector-design.md). +The new discussion proposes a 2.1 focus, USB/TCP first and MQTT forwarding; +the older possible 3.0 release pairing below is historical. The preview still uses +the verified HTTPS path and its existing radio lease. No MQTT/control/firmware +change is authorized merely by the discussion. The deployed app revisions are +recorded in the [current audit](post-20-audit-20261003.md). + Requested 2 October 2026. Work stays experimental on `n30nex-test`, based on upstream server `af20beb` / web `0924260` plus Atlas, Topology and exact-route evidence. Review requests and the daily automation remain on hold. From a17123af9f8e839b9ae3339ee78d2409487fb187 Mon Sep 17 00:00:00 2001 From: MrAlders0n <55921894+MrAlders0n@users.noreply.github.com> Date: Sat, 3 Oct 2026 20:06:10 -0400 Subject: [PATCH 59/69] chore: add .github/CODEOWNERS for main branch policy Signed-off-by: MrAlders0n --- .github/CODEOWNERS | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 181841b..d369ebd 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -1,2 +1,2 @@ -# Default owners for everything -* @446564 +# Every change to main needs a review from the repo owner +* @MrAlders0n From 5a60f1e00b3e7c13c416382d4a53f4ea84b7b0f4 Mon Sep 17 00:00:00 2001 From: MrAlders0n <55921894+MrAlders0n@users.noreply.github.com> Date: Sat, 3 Oct 2026 20:06:12 -0400 Subject: [PATCH 60/69] chore: add .github/workflows/main-source-guard.yml for main branch policy Signed-off-by: MrAlders0n --- .github/workflows/main-source-guard.yml | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) create mode 100644 .github/workflows/main-source-guard.yml diff --git a/.github/workflows/main-source-guard.yml b/.github/workflows/main-source-guard.yml new file mode 100644 index 0000000..911186f --- /dev/null +++ b/.github/workflows/main-source-guard.yml @@ -0,0 +1,24 @@ +name: Main PR source guard + +# Runs from main's copy of this file, so a PR can't edit the check away. +on: + pull_request_target: + branches: [main] + types: [opened, reopened, synchronize, edited] + +permissions: {} + +jobs: + main-source-is-dev: + name: main-source-is-dev + runs-on: ubuntu-latest + steps: + - name: Require PR from this repo's dev branch + env: + HEAD_REF: ${{ github.event.pull_request.head.ref }} + HEAD_REPO: ${{ github.event.pull_request.head.repo.full_name }} + run: | + if [ "$HEAD_REF" != "dev" ] || [ "$HEAD_REPO" != "$GITHUB_REPOSITORY" ]; then + echo "::error::PRs into main must come from $GITHUB_REPOSITORY:dev (got $HEAD_REPO:$HEAD_REF)" + exit 1 + fi From 29113a652e7195abd8ef362595835e587010316f Mon Sep 17 00:00:00 2001 From: n30nex Date: Sun, 4 Oct 2026 10:30:58 -0400 Subject: [PATCH 61/69] [Torchlight] docs: update roadmap and preview checkpoint [skip ci] Agent: Torchlight --- docs/meshat-review-20261004.md | 117 +++++++++++++++++++++++++++++++++ docs/post-140-roadmap.md | 77 +++++++++++++++++++++- 2 files changed, 193 insertions(+), 1 deletion(-) create mode 100644 docs/meshat-review-20261004.md diff --git a/docs/meshat-review-20261004.md b/docs/meshat-review-20261004.md new file mode 100644 index 0000000..70f1401 --- /dev/null +++ b/docs/meshat-review-20261004.md @@ -0,0 +1,117 @@ +# Meshat review candidates for Beacon + +Status: review options only, 4 October 2026. Adding an item here is not implementation +approval, an assigned task or a promised release. Return to the [project roadmap](post-140-roadmap.md). + +## Baseline and evidence + +The supplied Meshat report describes archived v1 at +[`f4c505b`](https://github.com/Bjorkan/meshat-beacon/tree/f4c505b07a2b4a47ddf6b90a0756968d795741f5). +Its [upstream integration record](https://github.com/Bjorkan/meshat-beacon/blob/f4c505b07a2b4a47ddf6b90a0756968d795741f5/beacon-docs/UPSTREAM_SYNC.md) +separates imported Beacon capabilities from retained fork work. The report's +39 headings include supporting API/migration/PR summaries, not 39 independent gaps. + +The comparison used released server +[`0015430`](https://github.com/MeshCore-Beacon/beacon-server/tree/001543032f4da4c532b59491f643f4e6911c4011) +and web +[`7a9770e`](https://github.com/MeshCore-Beacon/beacon-web/tree/7a9770e71399d1235359802b046111876e547d6c) +(v2.0.0), plus the v2.0.1 deltas: server `0dca03c`, web `7fb2478`. +It is a source-level review, not a live/dev deployment audit. The reported CoreScope +user migration and fork benchmarks are not independently verified causal/performance evidence. + +## Review first + +- **Realtime correctness:** inspect global broadcast loss, per-write WS deadlines and + Retry-After shared state. Released `Broadcast` logs/drops before fan-out when its + queue is full; client-buffer lag notices are not the same case. `noteRequestOk` + clears the shared rate-limit state on any successful response. These are candidates + for focused reproduction/regressions, not claimed production incidents. +- **Resolved list-hop names:** bounded optional REST packet/backfill and trace-summary + enrichment, preserving raw hashes/confidence and avoiding one detail fetch per row. + Existing detail/WS resolution and captured endpoint snapshots are not absent. +- **Current IATA-membership freshness:** distinguish current membership from historical + reception. Last-heard metadata alone is not an expiry policy for an otherwise active node. +- **Contact QR/deep links:** a small client-side MeshCore handoff candidate. Validate + full keys/name/type, generate locally and qualify real supported clients. + +## Larger choices and dependencies + +- **Global collision/confidence semantics:** released resolution/reconfirmation is + IATA-scoped; existing confidence labels and breaks at unmappable hops already help. + Decide how global ambiguity and useful contextual diagnostics coexist. +- **Directional neighbor SNR and provenance:** replace last-valid-value dependence + with bounded samples, counts, age and explicit receive direction where justified. + Separate fresh direct confirmation from inferred edges. An observer's terminal + receive SNR is not every hop's SNR; opposite directions are not interchangeable. +- **Calculated route planner:** weighted best/alternative graph routes and export are + additional to known-route search/cross-IATA composition. Define confidence, + direction, freshness, bounded computation and stale-snapshot behavior first. + Retain observed-route history; a proposed route is not proof of RF reachability. +- **Transit-node packet history:** additional to originating-node observations. Needs + indexed retained evidence, ambiguity-aware attribution and separate TRACE semantics. +- **Retained-history search:** broader path/payload/observer search, not only loaded + client rows. Bound/index it and disclose expired evidence. +- **Generalized server-side sorting/keysets:** cursor paging already exists; extend + full-dataset ordering, tie-breakers and filter-bound cursors where required. + +Dependencies: confidence/provenance and directional freshness precede a trustworthy +planner; indexed evidence precedes traversal/search; server ordering precedes a UI +promise of globally sorted results. Sizes and implementation owners remain undecided. + +## Optional or targeted improvements + +Consider API contract generation/drift checks, app-lifetime WS-to-query cache policy, +URL-owned list filters and intent preloading as focused maintainability work rather +than mandatory framework migration. Lightweight node mini-maps, radio-setting titles, +scoped unknown-channel totals and semantic mobile sorting need concrete user benefit. +A firmware-reported MeshCore region model is separate from IATA geography and transport +scope. Privileged observer-owner ingestion needs an explicit producer/privacy contract. + +Observer expiry is not wholly missing: released Beacon already has opt-in cleanup +that preserves observers referenced by retained observations, telemetry or ownership. +Meshat's snapshot-before-delete policy is a separate lifecycle decision. Treat its +visual/a11y/runtime audit as reproducible test cases, not dozens of proven current bugs. + +## Already covered or unsuitable for wholesale import + +Released Beacon already has the MapLibre 6 missing-image resolver, pending-region +guards, reconnect-specific heartbeat state, React Query, virtualization, lazy heavy +views and localization. Swedish is in v2.0.1. It also wires and matches WS route-type +and observer filters: importing Meshat's removal would discard working functionality. +View on map already exists; a node mini-map and MeshCore contact handoff are separate. + +Do not copy the whole router/Radix/DataTable/marker stack, Swedish deployment defaults, +single-broker restriction, a hard 150 km RF cutoff, fixed seven/fourteen-day retention, +or pre-2.0 migration numbers. Preserve useful uncertainty, unknown/zero distinctions, +historical evidence, license/author attribution and deployment-specific choices. + +## Reconcile against the current preview before implementation + +The [4 October preview checkpoint](post-140-roadmap.md#current-preview-checkpoint-4-october-2026) +now documents TRACE wire validity, conflicting identity handling, suspect-observation +filtering and protection against promoting requested/unvisited hops into observed +routes. Recheck that source before treating every Meshat TRACE/collision item as a +remaining preview gap. Ordinary-path resolution, full global confidence policy and +rolling directional link quality remain separate review topics. + +The new bounded Topology link snapshot is an undirected adjacent-pair read model, +not Meshat's weighted route planner or proof of radio delivery. Public chatter is +not general channel analytics. Compact Atlas telemetry is not complete node/fleet +history. Experimental delivery does not establish upstream release acceptance. + +## Evidence anchors and selection gate + +- [Released routing handlers](https://github.com/MeshCore-Beacon/beacon-server/blob/001543032f4da4c532b59491f643f4e6911c4011/internal/api/handlers/routes.go) and + [node handlers](https://github.com/MeshCore-Beacon/beacon-server/blob/001543032f4da4c532b59491f643f4e6911c4011/internal/api/handlers/nodes.go). +- [Neighbor/membership/retention queries](https://github.com/MeshCore-Beacon/beacon-server/blob/001543032f4da4c532b59491f643f4e6911c4011/db/sqlc/queries.sql.go). +- [Hub filtering/overflow](https://github.com/MeshCore-Beacon/beacon-server/blob/001543032f4da4c532b59491f643f4e6911c4011/internal/hub/hub.go), + [WS handler](https://github.com/MeshCore-Beacon/beacon-server/blob/001543032f4da4c532b59491f643f4e6911c4011/internal/ws/handler.go) and + [client rate-limit state](https://github.com/MeshCore-Beacon/beacon-web/blob/7a9770e71399d1235359802b046111876e547d6c/src/api/rate-limit.ts). +- [Existing MapLibre resolver](https://github.com/MeshCore-Beacon/beacon-web/blob/7a9770e71399d1235359802b046111876e547d6c/src/features/map/useMapLibre.ts). +- [Meshat planner API](https://github.com/Bjorkan/meshat-beacon/blob/f4c505b07a2b4a47ddf6b90a0756968d795741f5/beacon-server/internal/api/handlers/routes.go) and + [additional node APIs](https://github.com/Bjorkan/meshat-beacon/blob/f4c505b07a2b4a47ddf6b90a0756968d795741f5/beacon-server/internal/api/handlers/nodes.go). + +Before selecting work: verify the current target branch and existing issues/PRs, +identify remaining preview versus upstream gaps, define the counting/evidence and +compatibility contract, agree bounded acceptance tests, and assign an owner. Nothing +in this brief starts implementation or removes existing release/automation holds. diff --git a/docs/post-140-roadmap.md b/docs/post-140-roadmap.md index 4b6ec27..beeea1d 100644 --- a/docs/post-140-roadmap.md +++ b/docs/post-140-roadmap.md @@ -1,6 +1,81 @@ # Beacon roadmap: 2.0 integration and CoreScope parity -## Current: released 2.0 base and verified preview — 3 October 2026 +## Current preview checkpoint, 4 October 2026 + +The [Canadaverse preview](https://canadaverse.org/beacon-dev/) now serves web +**2.2.1-n30nex.1**, source +[`c41e6fa8`](https://github.com/n30nex/beacon-web-contributions/commit/c41e6fa8940fba929704bc2f9a94f119fef4a4ca), +and server **2.2.0-n30nex.1**, source +[`84ac3c87`](https://github.com/n30nex/beacon-server-contributions/commit/84ac3c8781ed19f99e7d996e07e8b671dc6efc92). +Public revision/index checks and current `n30nex-test` heads matched on 4 October. +Collector revision remains `aa7ddf06`. These are experimental preview versions, +not upstream release acceptance or a change to the proposed release priorities. + +Delivered in this preview, according to the corresponding source and +[public changelog](https://canadaverse.org/beacon-dev/source.html): + +- Atlas has compact aligned cards, expandable environmental telemetry and known + neighbours inferred from received packets, without new radio polling. The latest + patch reserves the telemetry alignment height only at desktop widths, labels + unavailable observer telemetry and keeps compact temperature channel labels readable. +- Shared sparklines distinguish measured segments, dashed gap estimates and single + readings. Missing telemetry is not fabricated; role badges and status styling are + consistent across views. +- Topology Public chatter is bounded to two bubbles, one on phones, and can be + disabled. Public identity comes from the actual decryption key rather than a + one-byte hash/name. Duplicate/private messages and ambiguous anchors are excluded + by the documented behavior; receiver pointers are not claims about the sender's location. +- `GET /api/v1/routes/topology` supplies unique adjacent, undirected known-route + node-ID pairs for 15m/1h/24h windows. Source caps replies at 100,000 links, reports + capping/window bounds, uses a 30-second cache and retains API rate limits. It avoids + route-page fan-out, not the need to distinguish observed evidence from RF reachability. +- TRACE handling distinguishes 1/2/4/8-byte hashes from ordinary 1/2/3-byte paths, + retains conflicting candidates and all eight bytes for eight-byte matching, and + reconstructs original SNR bytes. Unsupported/malformed/ambiguous observations are + hidden by default with a retained-evidence inspection option and diagnostic reasons. + Suspect or unvisited requested hops cannot become usable mapped/observed routes; + malformed traces do not create new neighbour/capability evidence. Raw history remains. +- The compact shared-region Topology controls, Changelog access and Torchlight's + Traces row-height, long-path containment and expand/collapse work remain preserved. + +Verification is deliberately bounded. Torchlight checked public revisions/index, +reviewed the latest web/server commits and inspected fixed desktop/mobile Packets +and Topology captures. Those captures reported no browser errors or failed responses. +The default page was Packets, not Atlas: Atlas card interactions, Public chatter +eligibility, the 24-hour switch, collision-control interactions and mobile-menu +behavior are not claimed browser-verified by this review. The changelog reports +native build/lint and **1,274 frontend tests passed**; these were not rerun here. +Earlier validation totals and deployment statements below belong to their dated +checkpoints. Keep the prior expanded-Trace-report visual acceptance gap explicit. + +## Meshat review queue, not implementation approval + +The [Meshat decision brief](meshat-review-20261004.md) adds the supplied fork review +to this roadmap. Its baseline is released Beacon 2.0.0 with the 2.0.1 deltas checked, +not this experimental preview. In particular, the new preview TRACE safeguards now +overlap parts of the collision/protocol review; recheck that delta before opening +implementation work. A bounded Topology link snapshot is not a weighted route planner. + +- **Review first:** global WS-loss notices, write deadlines and shared Retry-After + state; bounded resolved list-hop names; current IATA-membership freshness; contact + QR/deep links. Reproduce source-level candidates on the target revision first. +- **Design before scheduling:** global confidence semantics and directional SNR/ + evidence freshness, then a possible calculated route planner. Transit-node history, + retained-history search and generalized sortable keyset paging remain distinct choices. +- **Optional:** API contract drift checks, shared live-cache policy, richer URL-owned + filters, lightweight node mini-maps, radio titles, unknown-channel totals and + privacy-reviewed region/owner metadata integrations. +- **Do not duplicate or blindly import:** already-covered MapLibre resolver, region + pending-state and heartbeat mechanisms; working WS route/observer filters; Swedish + support in 2.0.1; wholesale framework changes, deployment-specific broker/branding + defaults, fixed RF/retention policies or pre-2.0 migration sequences. + +These entries remain review candidates. They do not approve a feature, supersede +an accepted priority, assign an implementation owner or authorize schema/RF changes. +The existing phase list is retained as planning history; decisions need explicit +scope, evidence, acceptance criteria and ownership before execution. + +## Historical checkpoint: released 2.0 base and verified preview, 3 October 2026 The [Pi preview](https://canadaverse.org/beacon-dev/) runs server **8c7fbb8** and web **7e139d5**, retaining My Atlas, Topology and node/route work on top of the From 92ba5b34918ae9c22da164707a3d6b277b5e567b Mon Sep 17 00:00:00 2001 From: n30nex Date: Sun, 4 Oct 2026 11:41:41 -0400 Subject: [PATCH 62/69] [Torchlight] docs: classify post-2.0 Meshat and preview candidates [skip ci] Agent: Torchlight --- docs/post-140-roadmap.md | 30 +- docs/post-20-feature-decisions-20261004.md | 330 +++++++++++++++++++++ 2 files changed, 356 insertions(+), 4 deletions(-) create mode 100644 docs/post-20-feature-decisions-20261004.md diff --git a/docs/post-140-roadmap.md b/docs/post-140-roadmap.md index beeea1d..527a9b2 100644 --- a/docs/post-140-roadmap.md +++ b/docs/post-140-roadmap.md @@ -1,5 +1,27 @@ # Beacon roadmap: 2.0 integration and CoreScope parity +## Post-2.0 decision register, 4 October 2026 + +The [unified Meshat and preview decision register](post-20-feature-decisions-20261004.md) +is the current review index. It reconciles all three upstream core repositories' +`main`/`dev` refs with Meshat v1 and the 2.2.1 preview, including existing PRs. + +- **✨ Review first:** realtime/API correctness, TRACE evidence safety and list usability, + focused My Atlas, bounded Topology reads, truthful telemetry, contact handoff and + current IATA-membership freshness. +- **❕ Design first:** directional evidence/aging/global ambiguity, calculated routes, + transit/history/search/sorting, targeted contract/cache work, Collector readiness + and optional map/chatter refinements. +- **❌ No new port:** already-covered mechanisms, duplicate contributions, whole-fork + rewrites, deployment-specific defaults and unjustified RF/retention policies. + +Use **🟢 KEEP / EXISTS**, **🟡 INTEGRATE / VERIFY / DESIGN**, and **🔴 DO NOT PORT / +DUPLICATE**, always with the reason. Worth keeping is not proof of upstream acceptance +or full stability. The register supplies stable R/D/X IDs, sources, dependencies, +B1–B9 corrections and acceptance gaps. Recommendations do not approve feature work, +assign owners/releases or remove existing holds. Prior checkpoints and planning phases +below remain preserved; use their dates rather than treating them as current verification. + ## Current preview checkpoint, 4 October 2026 The [Canadaverse preview](https://canadaverse.org/beacon-dev/) now serves web @@ -164,7 +186,7 @@ list is separated from release acceptance and production-load qualification. | **Hash and prefix tools** | Collision/usage matrix, role-aware prefix-width analysis and a prefix checker with explicit ambiguity | [Hash and prefix analysis](https://github.com/Kpa-clawbot/CoreScope/blob/093e320c2bda99d1fef317d7fc21fc1240a8cd12/public/analytics.js). Beacon already displays width distributions and ambiguous candidates; it lacks the dedicated tools. | | **Distance and route patterns** | Valid-coordinate hop/path distances, signal-versus-distance views, common subpath rankings and evidence-linked route alternatives/inspection | [Distance and route-pattern analysis](https://github.com/Kpa-clawbot/CoreScope/blob/093e320c2bda99d1fef317d7fc21fc1240a8cd12/public/analytics.js). Existing route search, detail and 3D layout do not supply geographic-distance analytics. | | **Historical replay** | Play/pause/seek, stepping and speed controls over retained observations; one time controller for map and Topology, with visible gaps | [Live/VCR guide](https://github.com/Kpa-clawbot/CoreScope/blob/093e320c2bda99d1fef317d7fc21fc1240a8cd12/docs/user-guide/live.md#vcr-mode). Beacon's live pause and static route-history windows are not replay. Hourly rollups cannot recreate expired packet paths. | -| **Geographic area filtering** | Filter nodes and their attributed traffic by advertised location/polygon across views, separately from receiving-IATA groups | [Area filter](https://github.com/Kpa-clawbot/CoreScope/blob/093e320c2bda99d1fef317d7fc21fc1240a8cd12/docs/user-guide/area-filter.md). Beacon's IATA groups, boundary overlay and optional foreign-node classification do not provide this complete workflow. | +| **Geographic area filtering** | Filter nodes and their attributed traffic by advertised location/polygon across views, separately from receiving-IATA groups | [Area filter](https://github.com/Kpa-clawbot/CoreScope/blob/093e320c2bda99d1fef317d7fc21fc1240a8cd12/docs/user-guide/area-filter.md#area-filter). Beacon's IATA groups, boundary overlay and optional foreign-node classification do not provide this complete workflow. | Smaller parity items remain below those analysis workflows: node/channel QR sharing, more table/layout preferences, an in-browser theme editor with import/export, and @@ -177,7 +199,7 @@ Audio/Lab and decorative exhibition modes are optional, not blockers for the cor analysis roadmap. Account synchronization/MeshMapper login is a separate ownership and authentication decision, not a prerequisite for browser-local Atlas. -## Delivery order +## Delivery order (earlier planning context) | Phase | Next package | Completion gate | |---|---|---| @@ -189,11 +211,11 @@ and authentication decision, not a prerequisite for browser-local Atlas. | **5 — history and geography** | Retained packet replay, observer reach/timing and GPS-area filtering | Ordered bounded cursors, explicit retention/gaps, no invented paths or clock-based claims of RF propagation. | | **Beyond parity** | Evidence-linked MeshMapper scope/boundary crossings, region changes and guided live follow | Distinguish a receiving-IATA change, advertised location, scope label and a geometric crossing; never manufacture neighbour links from catalogue counts. | -**Recommended next new implementation after 2.0:** the node dashboard/trace phase. +**Earlier recommended next new implementation after 2.0:** the node dashboard/trace phase. It builds on the existing node inspector and Atlas instead of adding another independent page with different counts. Replay is the largest remaining live-view capability gap and needs its retained-evidence contract before UI implementation. -This update schedules work; it does not start new feature implementation. +This earlier order is planning context, not approval to start feature implementation. ## CartoLite-derived follow-ups diff --git a/docs/post-20-feature-decisions-20261004.md b/docs/post-20-feature-decisions-20261004.md new file mode 100644 index 0000000..97486fb --- /dev/null +++ b/docs/post-20-feature-decisions-20261004.md @@ -0,0 +1,330 @@ +# Post-2.0 Beacon: Meshat and preview decision register + +[Roadmap](post-140-roadmap.md) · [Earlier Meshat review](meshat-review-20261004.md) + +Prepared by **[Torchlight]**, 4 October 2026, for the Beacon team. +[Request and colour convention](https://discord.com/channels/1507764602253869197/1507788454774182030/1556327984896675861). +This is a recommended review order, not approval to implement every item, a release +schedule, or permission to change production, databases or radios. Existing accepted +priorities and draft/review/automation holds remain in force. + +## Read the labels correctly + +- **✨ Review first:** clear potential benefit; select a bounded upstream contribution. +- **❕ Design first:** meaningful capability, but evidence, cost or product decisions remain. +- **❌ No new port:** already covered, duplicate, unsuitable default or outside this scope. +- **🟢 KEEP / EXISTS:** worthwhile or already part of Beacon. This is not a blanket stability claim. +- **🟡 INTEGRATE / VERIFY / DESIGN:** implementation, validation or a decision remains. +- **🔴 DO NOT PORT / DUPLICATE:** reject the proposed addition, with a reason. Use BROKEN only + for an evidenced defect; do not label an existing working module broken because its duplicate is red. + +IDs below are stable planning identifiers. Source state, recommendation and delivery +state are separate. None of the new candidates below is marked accepted or shipped +upstream merely because this document recommends it. + +## Current upstream and preview baseline + +Refs were checked on 4 October 2026, approximately 15:32–15:36 UTC: + +- **Server:** main `3828e689eda947391ca35c51934a7e79c9b40a82`; + dev `c7209b70433b8b127a5b1062fdfb17d4a676245c`. + Main is v2.0.1 `0dca03cd4dae1b6061df5078a4390be33b9ed475` plus branch-governance files. + The inspected main/dev comparison adds governance, not a new feature release. + Catalogue-refresh fairness and the default 24-hour refresh are already included. +- **Web:** main `82fb6835066aed7acc3f611cab450b3760d08fa3`; + dev `b4d498e697182f83bef47152c3f35afb226a2252`. + Main is v2.0.1 `7fb24789d1649aa079e4568d335d60b3400d87be` plus governance. + Dev additionally includes [#146](https://github.com/MeshCore-Beacon/beacon-web/pull/146), + the mobile channel-scroll/bottom-navigation fix (`dc757cfb5147108e6b0ebbb4adc4b14e07a9f192`). + Divergent release history makes the raw ahead/behind count misleading; it is not a feature count. +- **Docs:** main `f0d5632ad9833305745f54ff1bb2c0e2bb413f8f`; + dev `5a60f1e00b3e7c13c416382d4a53f4ea84b7b0f4`. + The inspected dev difference is branch governance. The consolidated operator/API docs + already exist; an experimental roadmap is not an accepted upstream feature contract. +- **Canadaverse:** web **2.2.1-n30nex.1** at `c41e6fa8940fba929704bc2f9a94f119fef4a4ca`; + server **2.2.0-n30nex.1** at `84ac3c8781ed19f99e7d996e07e8b671dc6efc92`. + Public index/revisions were reverified. Collector revision is unchanged at + `aa7ddf06034620bb8c7c60b2011ce5bb98f313d2`; no private implementation was inspected here. +- **Meshat:** archived v1 `f4c505b07a2b4a47ddf6b90a0756968d795741f5`, not every later fork branch. + The supplied long report and Claude's follow-up are review evidence, not test results. + Claude's `v2.0.1 / origin/dev` wording does not pin an exact inspected dev commit. + +Coverage means current refs, recent history, relevant PRs, targeted source and retained +release review across the three public core repositories, not an exhaustive audit of +all code or every repository in the organization. The available recent channel context +and saved reviews were read; this is not a claim to have retrieved every channel message. + +## ✨ 1. Worth reviewing first + +### R01 — 🟢 KEEP; 🟡 integrate/test: realtime correctness (Meshat) + +Prioritize global hub-loss notification, authoritative Retry-After deadlines, bounded +WS writes and reliable recovery of affected views. Current source still logs/drops a +full global broadcast queue before client fan-out; `noteRequestOk` clears shared +backoff; `writeTimeout` remains unused. Client-buffer lag notices and heartbeat reset +already exist and are not substitutes for these fixes. + +Benefit: fewer silent live/history inconsistencies and less avoidable retry pressure. +Acceptance: queue-saturation regression, coalesced recovery without a refetch storm, +concurrent 429/200 ordering, slow-reader cancellation and shutdown cleanup. Preserve +working route/observer filters. Shutdown and all-view resync claims need separate tests; +no production incident or throughput improvement was reproduced in this review. + +### R02 — 🟢 KEEP; 🟡 targeted fixes: API and UI correctness (both) + +Apply documented region filters consistently to ordinary message lists; reject invalid +node-type names rather than silently removing the filter. Reuse preview error/retry +and missing-data patterns where a current upstream UI reproduces the same defect. +Audit pathological drift, storage-denied environments and detail error states rather +than accepting every old fork audit finding as current. + +Acceptance: slug/ID/IATA combinations, empty/unknown regions, valid aliases versus +invalid types, 404 versus 5xx, loading/no-data versus real zero, and storage exceptions. +The server already distinguishes missing records from server failures. Backfill already +resolves region filters. Sub-hour rollup semantics are a separate design item, not a +mechanical bug fix. See the B1–B9 reconciliation below. + +### R03 — 🟢 KEEP; 🟡 port with protocol tests: TRACE evidence safety (preview + Meshat) + +Strong upstream candidate: distinguish TRACE 1/2/4/8-byte hashes from ordinary +1/2/3-byte paths; preserve conflicting identities and all eight matching bytes; +reconstruct original SNR bytes; prevent malformed or unvisited requested hops from +becoming observed routes, neighbours or capability evidence. Explain suspect records +and retain access to raw evidence instead of deleting history. + +Acceptance: live/REST parity, signed SNR and reconstruction fixtures for each supported +width, malformed/truncated inputs, ambiguous candidates, requested versus visited hops, +and no new inferred links from invalid data. Default hiding and wording need usability +review. This is not a claim that all upstream TRACE handling is broken, nor that the +preview solves global ambiguity for every ordinary path. + +### R04 — 🟢 KEEP; 🟡 integrate: readable, bounded trace and packet lists (both) + +Keep preview row-height/long-path containment, visible expansion controls and bounded +collapsed previews. Add Meshat-style optional resolved hop summaries to list/backfill +responses where missing, retaining hashes and confidence instead of doing one detail +request per row. Those are two separable patches, not an excuse to rewrite all lists. + +Acceptance: long paths, click/Enter/Space, selection, expanded report width, populated +desktop/mobile captures, bounded batch resolution and consistent live/reloaded rows. +Earlier preview compact/mobile evidence exists; expanded-report and loaded desktop +acceptance were not completed by the latest fixed captures. + +### R05 — 🟢 KEEP; 🟡 upstream acceptance: focused My Atlas (preview) + +Saved full-key node monitoring, compact cards, optional environmental telemetry, +known-neighbour context and shared inspection links are useful post-2.0 capabilities. +Start from [existing draft #97](https://github.com/MeshCore-Beacon/beacon-web/pull/97), +not a duplicate implementation. Its head is `91a9541cd6fe27f76996c8f7e65fb07698783a98`, +not the latest preview; newer compact/telemetry changes need explicit reconciliation. + +Keep a browser-local, read-only core independent of Collector/control rollout. +Acceptance: refresh against current dev, storage migration/failure handling, full-key +identity, bounded samples, stale/error states, mobile/keyboard access and optional +telemetry service absence. A 200-report sample is not a complete node/fleet total. + +### R06 — 🟢 KEEP; 🟡 focused integration: Topology and bounded link reads (preview) + +Keep a focused optional Topology view with shared multi-region selection, camera state +preserved across refreshes, compact controls, lazy optional context and visible caps. +The 24-hour link endpoint is a particularly useful query-shaping candidate: one cached, +bounded read instead of a client fan-out across many route pages. + +The preview snapshot returns deduplicated adjacent **undirected** node-ID pairs for +15m/1h/24h, caps at 100,000 links and reports window/capping; prior source review found +30-second caching and a 15-second request timeout. Preserve rate limits. +Acceptance: indexed query plans and realistic load, cache/filter isolation, missing-hop +breaks, capped/stale labels, a paired API/UI rollout, bounded rendering and low-power +fallback. This is neither a weighted planner nor proof of RF reachability. No measured +speedup or production capacity is claimed here. + +### R07 — 🟢 KEEP; 🟡 reconcile existing components: truthful telemetry and charts (preview) + +Reuse compact optional telemetry, labelled unavailable readings, single-sample display, +consistent roles/status and clear measured-versus-gap styling. Upstream already has +hourly chart gaps; port only the missing behavior, not a second chart system. If dashed +segments estimate across gaps, label them as estimates and never count them as samples. + +Acceptance: absent/null/zero distinction, intermittent and single samples, units, +temperature-channel labels, dark/light themes, keyboard/mobile layout and bounded +refreshes. Coordinate palette/accessibility work with open +[web #147](https://github.com/MeshCore-Beacon/beacon-web/pull/147), not a competing rewrite. + +### R08 — 🟢 KEEP; 🟡 small independent candidates: contact handoff and radio names (Meshat) + +A locally generated MeshCore contact QR/deep link is a visible, contained improvement. +Validate full keys/name/type and actual supported clients. Friendly radio-preset names +are useful secondary enrichment: bundle/version the catalogue, retain raw settings and +show an honest fallback for no match or ambiguity. Do not require a startup network fetch. +Neither a QR nor a preset label proves device compatibility or radio reachability. + +### R09 — 🟢 KEEP; 🟡 agree policy: current IATA-membership freshness (Meshat) + +Distinguish where a node was ever heard from current membership of an otherwise-active +node. `last_heard` metadata is not itself an expiry policy. Prefer a configurable read +policy with visible age over erasing historical receptions. +Acceptance: one-off distant reception, reappearance, inactive/active nodes, cutoff +boundaries, region filtering and query cost. The specific duration is undecided; it is +not automatically seven days. More invasive neighbour aging/mobility work is D01. + +## ❕ 2. Meaningful additions that need larger decisions + +- **D01 — 🟡 DESIGN: evidence quality, direction and ambiguity (Meshat + preview).** + Rolling directional SNR with sample count/age, direct versus inferred provenance, + edge aging and node-movement invalidation, and global versus IATA-contextual identity + confidence. Valuable even without a planner. Define provenance, freshness and migration + cost first; terminal receiver SNR is not every hop's SNR. Preserve uncertain diagnostics. +- **D02 — 🟡 DESIGN: calculated best/alternative routes (Meshat).** + Dijkstra/Yen-style routing and MeshCore export add to observed-route browsing/search. + Depends on D01 plus bounded graph refresh/query cost and stale-snapshot behavior. + Keep observed routes; a proposed path is not verified RF delivery. No planner is approved. +- **D03 — 🟡 DESIGN: investigations and retained history (both).** + Transit-node packet history, full node/fleet metrics, trace reception chronology, + indexed history/path/text search, observer filters and generalized server-side sorting. + Separate origin observations from candidate relay evidence; define indexes, retention, + deduplication, keyset ties, count cost and ambiguity. Build a cheap observer filter + separately from whole-history text search. Replay/seek and geography remain later + retained-evidence workflows, not features provided by a live map or hourly rollups. +- **D04 — 🟡 SELECT TARGETED WORK: contracts, cache and accessible navigation (Meshat).** + OpenAPI drift checks, a consistent app-lifetime WS/query reconciliation policy, + URL-owned filters and semantic mobile sort can help without replacing the router. + Prefer small measured improvements. API sorting precedes claims of globally sorted + results; focus/keyboard testing, not the presence of Radix, establishes accessibility. + Sub-hour statistics belong here as an explicit resolution/API decision when needed. +- **D05 — 🟡 RELEASE DESIGN: Collector and additional metadata (preview + Meshat).** + Optional telemetry acquisition needs supported transports, reconnect/long-run tests, + per-radio budgets, shared repeater leases, auth/revocation and a clear producer contract. + Core Atlas does not depend on approving remote radio control. Self-reported MeshCore + regions are distinct from geographic IATA groups. Observer-owner ingestion requires + explicit privacy/privileged-feed decisions. Public design only; no private code or + hardware access was part of this review. Preserve existing 2.1 proposals and later + console/control, BLE/browser and firmware qualification boundaries. +- **D06 — 🟡 OPTIONAL UX: Public chatter, mini-maps and map polish (both).** + Preview chatter can add context, but it is not a performance priority or channel + analytics. Keep it bounded, optional, privacy-aware and anchored to a receiver rather + than asserting sender location; identify Public by the actual decryption key. + Mini-maps, trace maps, SNR-coloured legs, legends, theme-following basemaps, translated + labels, hover preload, saved display preferences and unknown-channel totals need + concrete user benefit and measured cost. SNR styling depends on D01's truthful evidence. + +## ❌ 3. Already covered, unsuitable or out of scope + +### X01 — 🟢 existing Beacon capability; 🔴 duplicate port + +Keep, do not reimplement or remove: + +- Working WS route-type/observer filters, reconnect-specific heartbeat state and + client-buffer lag handling. They do not solve R01's distinct global-queue loss. +- Pending-region request guards, MapLibre missing-image resolver, existing live map, + neighbour graph, View on map, lazy heavy views, React Query and virtualization. +- Existing i18n including Swedish in 2.0.1, clipboard failure handling, zero-radio + formatting and bounded list limits. A separate mini-map/contact QR is not View on map. +- Opt-in observer expiry that preserves referenced history, server 404/500 distinction, + hourly rollups/gaps, retained-summary metadata and existing route/trace keyset fixes. +- Catalogue-refresh fairness/default interval in server 2.0.1 and the consolidated + upstream operator docs. MeshMapper stays a read-only Beacon integration, not a redesign target. +- Mobile scroll/navigation fix #146 is already merged into **dev**, not a new Meshat port. + Chart palette #147 is **open**, not shipped; contribute to its review rather than duplicate it. + +### X02 — 🔴 do not import as a package or default + +- The whole fork, a forced Router/Radix/DataTable/marker rewrite, a mandatory new + architecture for feature parity, or decorative features as release blockers. +- Swedish branding/default locale/root catalogue and single-broker restrictions. + Preserve Beacon's deployment configurability and multi-broker support. +- A universal 150 km rejection rule, blanket destruction of one-byte diagnostics, + automatic seven/fourteen-day policies or averaged opposite-direction SNR. + Geographic plausibility is evidence to interpret, not a universal RF law. +- Pre-2.0 migration numbers or edits to the released flattened baseline. Any new schema + needs its own reviewed upgrade/recovery design; none is authorized by this report. +- Promoting intended TRACE hops, catalogue counts, missing records or uncertain clock + deltas into observed links, packet-loss totals or RF propagation claims. +- Upstream/production deployment, remote console/radio control, private source exposure, + or unrequested automation changes. They are not side effects of adopting a roadmap. +- The claim that a particular feature caused CoreScope users to migrate. No independent + adoption/causal evidence or portable performance benchmark was supplied. + +## Claude B1–B9: current-source reconciliation + +These IDs belong to the supplied notes, not nine automatically approved fixes. + +- **B1 🟡 source-backed:** `Hub.Broadcast` still drops/logs a full global queue. + Reproduce and add bounded lag/recovery signaling; distinguish deliberate repeat shedding. +- **B2 🟡 source-backed:** `noteRequestOk` still clears active shared backoff. + Test concurrent 200/429 ordering; no live request storm was reproduced. +- **B3 🟡 targeted contract gap:** ordinary `/messages` and channel-message listing use + IATA parsing without region expansion. **🟢 `/messages/backfill` already resolves regions**; + the notes must not be generalized to every message endpoint. +- **B4 🟡 protocol check:** live ingest builds metadata from `packet.Path`, `PathLength` + and ordinary path helpers. The upstream contract explicitly notes TRACE path bytes + are SNR bytes. Raw SNR preservation is not itself a bug: compare interpreted hash + width/count and live/REST behavior with valid fixtures, including the preview delta. + This review did not execute that regression or prove complete preview repair. +- **B5 🟡 audit:** pathological clock drift/ranking remains a supplied candidate, not a + freshly reproduced defect. Check timestamp bounds, units and valid extreme cases. +- **B6 split:** **🟡 type validation:** current handler and `NodeTypeFromString` silently + map an unknown nonempty name to zero/no filter. **❕ sub-hour policy:** + `parseStatsWindow` deliberately truncates to UTC hours and documents empty sub-hour + results. Do not silently replace hourly-rollup semantics with raw-data scans. +- **B7 🟡 partially source-backed:** unused `writeTimeout` and connection-context writes + confirmed. App-wide hijacked-socket shutdown behavior still needs dedicated verification. +- **B8 🟡 audit/integration:** detail error-state and all-view lag recovery are useful + acceptance cases; remaining current upstream callers were not exhaustively re-audited. + Do not erase existing server 404/500 handling or preview error/retry improvements. +- **B9 🟡 audit:** storage-denied reads/writes need focused tests. Earlier review found + an unguarded initial App read; every listed current call site was not rechecked here. + +## Suggested packages and dependency gates + +1. **Correctness package:** R01/R02 plus isolated R03 regressions. Select individual + fixes; do not label all B1–B9 cheap or confirmed. Keep protocol and UI patches reviewable. +2. **Visible-benefit package:** focused Atlas R05, trace usability R04, truthful telemetry + R07 and optionally contact QR R08. Reuse existing drafts and components. +3. **Bounded topology package:** R06 API plus UI qualification, preserving caps and rate + limits. Broader D01 evidence semantics remain a separate design, not an excuse to + treat undirected adjacent pairs as directional quality measurements. +4. **Evidence/data-design package:** R09 and D01, then decide whether D03 investigation + or D02 routing gives users more value. D01 precedes trustworthy SNR maps/planning; + indexed retained evidence precedes transit/search/replay; server ordering precedes + global-sort UI promises. D05 has independent transport/privacy/release gates. + +These recommendations supplement, not silently replace, the proposed Atlas/Collector +focus and historical phase list. Implementation owners, estimates and release slots +remain unset. Use the stable IDs to record a future accepted decision and its tests. + +## Evidence, existing work and limits + +- Current [server main/dev comparison](https://github.com/MeshCore-Beacon/beacon-server/compare/3828e689eda947391ca35c51934a7e79c9b40a82...c7209b70433b8b127a5b1062fdfb17d4a676245c), + [web main/dev comparison](https://github.com/MeshCore-Beacon/beacon-web/compare/82fb6835066aed7acc3f611cab450b3760d08fa3...b4d498e697182f83bef47152c3f35afb226a2252), + and [docs main/dev comparison](https://github.com/MeshCore-Beacon/beacon-docs/compare/f0d5632ad9833305745f54ff1bb2c0e2bb413f8f...5a60f1e00b3e7c13c416382d4a53f4ea84b7b0f4). + Web comparison is merge-base/history based; recent dev commits and release-to-main + comparison, not its raw count, establish the reported post-release delta. +- Current source: [hub](https://github.com/MeshCore-Beacon/beacon-server/blob/c7209b70433b8b127a5b1062fdfb17d4a676245c/internal/hub/hub.go), + [WS writes](https://github.com/MeshCore-Beacon/beacon-server/blob/c7209b70433b8b127a5b1062fdfb17d4a676245c/internal/ws/handler.go), + [ingest metadata](https://github.com/MeshCore-Beacon/beacon-server/blob/c7209b70433b8b127a5b1062fdfb17d4a676245c/internal/ingest/packet.go), + [Retry-After state](https://github.com/MeshCore-Beacon/beacon-web/blob/b4d498e697182f83bef47152c3f35afb226a2252/src/api/rate-limit.ts). +- Current source: [messages](https://github.com/MeshCore-Beacon/beacon-server/blob/c7209b70433b8b127a5b1062fdfb17d4a676245c/internal/api/handlers/messages.go), + [channel messages](https://github.com/MeshCore-Beacon/beacon-server/blob/c7209b70433b8b127a5b1062fdfb17d4a676245c/internal/api/handlers/channels.go), + [stats windows](https://github.com/MeshCore-Beacon/beacon-server/blob/c7209b70433b8b127a5b1062fdfb17d4a676245c/internal/api/handlers/regions.go), + [node handler](https://github.com/MeshCore-Beacon/beacon-server/blob/c7209b70433b8b127a5b1062fdfb17d4a676245c/internal/api/handlers/nodes.go), + [type conversion](https://github.com/MeshCore-Beacon/beacon-server/blob/c7209b70433b8b127a5b1062fdfb17d4a676245c/internal/api/nodes.go), + and [upstream API contract](https://github.com/MeshCore-Beacon/beacon-docs/blob/f0d5632ad9833305745f54ff1bb2c0e2bb413f8f/docs/api-contract.md). +- [Meshat archived source](https://github.com/Bjorkan/meshat-beacon/tree/f4c505b07a2b4a47ddf6b90a0756968d795741f5), + [original channel review](https://discord.com/channels/1507764602253869197/1507788454774182030/1556287124452671621), + [Claude notes](https://discord.com/channels/1507764602253869197/1507788454774182030/1556325720685281312), + [preview changelog](https://canadaverse.org/beacon-dev/source.html), + [pinned preview web](https://github.com/n30nex/beacon-web-contributions/tree/c41e6fa8940fba929704bc2f9a94f119fef4a4ca), + [pinned preview server](https://github.com/n30nex/beacon-server-contributions/tree/84ac3c8781ed19f99e7d996e07e8b671dc6efc92). +- [Server draft #192](https://github.com/MeshCore-Beacon/beacon-server/pull/192) remains + open at the newer preview head; its body still quotes older `8c7fbb8`/`7e139d5` + runtime evidence. Do not treat those old test totals as validation of every new change. + [Docs draft #7](https://github.com/MeshCore-Beacon/beacon-docs/pull/7) follows the owned + docs fork; neither it nor Atlas #97 is upstream acceptance. No reviewer request is made. + +No application build/test, load benchmark or fresh browser interaction test was run +for this report. The preview changelog reports 1,274 frontend tests; that is attributed, +not rerun evidence. Earlier screenshots covered Packets/Topology and populated mobile +Traces; desktop Traces captured loading skeletons, Atlas/chatter interactions and +expanded-report geometry remain acceptance gaps. Public HTTP health does not prove a +healthy whole data pipeline. All absent-feature statements are scoped to the inspected +interfaces and retained pinned review; larger audits remain explicit follow-up work. From e5cf64afa85275f7e19a754c9df9457cc95ac76e Mon Sep 17 00:00:00 2001 From: n30nex Date: Sun, 4 Oct 2026 14:17:04 -0400 Subject: [PATCH 63/69] =?UTF-8?q?[Torchlight]=20Adopt=20the=20primary=20Be?= =?UTF-8?q?acon=202.1=E2=80=932.2=20roadmap=20[skip=20ci]=20(#3)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ROADMAP.md | 441 +++++++---------------------- docs/beacon-21-collector-design.md | 10 + 2 files changed, 115 insertions(+), 336 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index 5664f09..27fc17b 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,336 +1,105 @@ -# Beacon parity and analytics roadmap - -## Current: released 2.0 baseline and experimental preview — 3 October - -The Pi preview includes released server `0015430` and web `7a9770e`, plus My Atlas, -Topology, full node pages and exact route evidence. Its deployed revisions are -server **8c7fbb8** (`2.1.0-n30nex.2`) and web **7e139d5** (`2.1.1-n30nex.1`). -Both contain the checked upstream main/dev branches. The separate private Collector -runs alpha.4; its implementation and prototype history are absent from public Beacon. - -The new 001 database baseline is active; the old database and verified recovery -copies remain intact. At the user's request, preview retention matches the released -defaults: **7 days raw, 90 days summaries, 31 days observer telemetry and 14 days -routes**. This replaces older preview-policy notes below. No production service, -configuration or database was changed. - -753 server tests and 1,259 frontend tests passed, plus native PostgreSQL/build/lint, -31 preview API checks and exact public assets/source checks. Topology camera -continuity, node/telemetry error handling and missing-number validation are fixed. -Deployment-log rotation is prepared in these templates under docs issue #13. - -The latest discussion proposes **Atlas and Collector as the 2.1 focus**. The -[planning scratchpad](docs/beacon-21-collector-design.md) compares MeshCore HA's -traffic controls and records USB/TCP, per-radio budgets, shared repeater leases -and MQTT decisions. Topology remains in the preview; full node/fleet history, -graphical trace analysis, shared search, channel analytics and replay remain on the -broader parity roadmap. Collector wider-alpha gates remain explicit. Experimental PRs stay -draft with no review pings; daily automation remains paused. Older checkpoints below -are historical, including their version, retention and recovery instructions. - -[Audit and remaining gates](docs/post-20-audit-20261003.md) · -[Parity roadmap](docs/post-140-roadmap.md) · -[Current experimental preview](docs/n30nex-test-preview.md). - -## Recorded 1.4.0 checkpoint — superseded on the Pi by the experiment - -The requested refresh now includes Alderson's latest accepted server **14354b03** and web **e2d272e0**, plus our release PRs. At that checkpoint, the Pi review site ran **server 61b0322a / web 157525ef**, from server #189 and web #105. This supersedes the earlier #107 cutoff. Web #108–#111 provide the shared observer sidebar, labelled device details, unified packet observations and removal of the duplicate Observer page in Analytics. The server includes the partial-telemetry and counter-bucketing corrections. - -**Beacon/web remains 1.4.0**, replacing the planned 1.3.2 release. My Atlas #97 is excluded until after 1.4.0 and still needs conflict resolution against the accepted release head. Alderson controls acceptance, stable tags and the production switch from CoreScope at `live.meshcore.ca`; `dev.meshcore.ca` remains development-only. Neither official host was changed. Server versions remain independent. - -Current-base checks and published-head CI pass. Native Go/PostgreSQL tests, restored-copy migration 045, the 3,200-input replay, and Windows/Pi web build/lint/**1,035 tests** pass. All **21 public assets**, both sources, **26 boundaries**, live packet delivery and desktop/French phone checks pass. The test-helper readiness race was fixed by draining probes through a unique marker; 100 repetitions of each affected test passed. Live ingestion behavior is unchanged by that helper fix. Existing image-tag and Compose receipts remain applicable to unchanged workflow/template content. - -Migration 045 removed **485 partial telemetry rows on the restored copy**, preserving raw counts, retained telemetry and node fingerprints. Immediately before live migration, **489 matching rows** were separately saved in the private checkpoint. The full restored/checksummed dump and this row export are retained on and off the Pi. - -Backend recovery is `evidence/sync-140-20260930/deploy-server.py rollback`, checkpoint `sync140-cutover-20260930T204913Z`. It restores **9054acd8 / 23945d59**, rolling back the new frontend first when needed. It preserves new traffic and the compatible telemetry cleanup; the private row export retains deleted rows for selective recovery. Frontend-only recovery is `evidence/sync-140-20260930/deploy-beacon-web.py rollback --evidence-dir sync-140-20260930`, checkpoint `web-20260930T211410Z`. Configuration remains b3d96f52 / mode 0644. The server update preserved the other 23 containers; the frontend preserved all 24. Automatic zones, public admin/backup and foreign classification stay disabled. - -**Remaining owner gates:** review/acceptance, the requested 24h/3d raw-history controls versus upstream 7d/30d controls, physical Safari, production-host capacity/data verification, approved Actions images/tags and the CoreScope switch. The bounded replay and 40-second live sample do not establish production capacity. - -[Current release/cutover plan](docs/release-140-preparation.md) · [Exact candidate record](docs/release-140-heads.json). - -## Historical 1.3.2 preparation - -## 1.3.2 release preparation — 30 September - -The Pi now serves exact merged dev server `689bc232` / web `00d859d9`, with Atlas excluded. The web stack landed through #99, followed by #100–#103. Server #182 and the maintainer ingestion, caching and location fixes are included. Native PostgreSQL tests, the 3,200-input replay, and web build/lint/all **1,017 tests** pass. All 21 public assets, both source archives and 26 boundaries match. - -My Atlas #97 remains held at `66ae0cc2` for after 1.3.2 and needs conflict resolution against the new dev branch. Web #93 is closed without merge. Upstream #101 restored 7d/30d controls and maps old 3d observer links to 7d; this conflicts with the requested 24h/3d raw-history controls and remains a release follow-up. Migration 043 was restore-tested: 273 stale location records cleared, raw counts and other located nodes preserved. The prior server/frontend and a verified on/off-Pi dump remain recoverable. - -[Current audit, exact heads, validation and recovery](docs/release-132-preparation.md). [Live candidate and source](https://canadaverse.org/beacon-dev/source.html). Maintainers retain web acceptance, tags, version decisions and production rollout. Earlier dated records below are historical. - -Updated 29 September 2026 (Toronto). This is the working roadmap for n30nex's ongoing contributions toward CoreScope feature parity. Maintainers decide acceptance and merge order; deployment owners handle the production switch. - -Refresh GitHub issues, PR feedback and branch state before starting a phase. This document is a snapshot, and linked issues/PRs are the current source of truth. - -Release-check correction, 21 September UTC: the workflow now includes independent preview PRs in Status and Check, applies the same CI/head/fork/target requirements to them, and rechecks the prepared independent inputs before publication. This covers packet summaries #161 and map correction #61 without adding them to the ordered stacks. The backup CLI #160 retains its separate check. Twenty-three offline regressions cover these gates and the existing no-rebase/cache behavior. See [the contributor workflow](CONTRIBUTOR_WORKFLOW.md). - -The following sections preserve dated historical checkpoints. Use the 30 September release record above for current source, acceptance and rollback. - -## Historical Atlas preview — 29 September - -[Exact feature, validation and recovery](docs/my-atlas-20260929.md). - -[Web #97](https://github.com/MeshCore-Beacon/beacon-web/pull/97), `4fd4b0de`, is the single My Atlas feature PR requested by the contributor. It follows #95 at `e1133ab5` and closes [web #96](https://github.com/MeshCore-Beacon/beacon-web/issues/96) on acceptance. Earlier application PRs remain included; no parent was rebased for this feature. - -My Atlas sits in the desktop tab row and the phone More menu. Visitors save up to twelve full-key node identities, order and a 24h/3d window in this browser. Compact cards show reception bars, SNR/RSSI meters and server freshness; Heard by and statistics expand on demand. Search collapses on return visits. Node, observer/dashboard and exact packet/path investigation reuse the existing navigation. English and French ship together. - -Counts are explicitly the latest **200 retained origin-key reports per node**, filtered to the selected period. Companion requests and other identified-origin packets are included as well as adverts. This is not a complete node-traffic total. Heard by describes the latest loaded packet, not lifetime reach. Missing readings, expired details and incomplete samples remain visible; no radio-health or packet-loss score is invented. - -The Pi now runs unchanged server `a35cba1d` with web `4fd4b0de`. Windows and native Pi build/lint/all **960 tests** pass, along with the actual published-head CI (web CodeQL skipped). Desktop, French 390px phone, keyboard, persistence/order/removal, packet/observer links and dashboard Back checks pass. The public 19 assets and both source archives match, both MQTT feeds are connected, and all 24 container identities/restart counts are unchanged. Physical iPhone Safari and full production capacity remain separate release gates. - -[My Atlas preview](https://canadaverse.org/beacon-dev/?tab=MyAtlas) · [Changelog and source](https://canadaverse.org/beacon-dev/source.html). Frontend rollback restores `e1133ab5` with server `a35cba1d`, from `web-20260930T004937Z`. For an older backend rollback, restore this frontend first, then use the existing September 29 backend recipe; its guard intentionally rejects an unknown newer frontend. - -The contributor explicitly prioritized My Atlas for this phase. Next: refresh maintainer feedback and issues, then server #183 before optional MeshMapper boundaries. Broader issues and remaining parity work stay open. All eighteen application candidates are out of draft; maintainers retain acceptance, merges, stable releases and production cutover. - -## Earlier review corrections and history windows — 29 September - -All six server reviews are addressed in their existing PR sequence. The fixes restore analytics indexes, consolidate unmerged migrations, preserve current partial activity buckets, align cache windows, narrow the route index, keep manual scope priority and simplify channel insertion metadata. Independent server #184 fixes RFC3339 offsets; new web #95 follows #92 and matches time choices to retained data. Existing candidates remain included. - -The Pi runs server `a35cba1d` / web `e1133ab5`. All seventeen published application heads pass Check/CI (web CodeQL skipped); native Go/PostgreSQL and Windows/Pi web build/lint/all 940 tests pass. The restored-copy repair preserved raw rows and archive fingerprints, and rollback index restoration passed. A 3,200-input/504-scope replay had all expected rows/events and zero fixture drops. The one-minute live sample had no parser fallbacks, queue overflows, SQL errors, restarts or reconnects; malformed-IATA and clock-skew warnings remain. - -Visible periods are **24h / 3d** for observer monitoring and route evidence, and **24h / 3d / 30d** for summary-backed Analytics. Seven-day buttons are removed. Raw comparison spans are capped at three days. Durable hourly aggregates already preserve expired packet counts; materialized views combine them with live rows. Older summaries accumulate after archiving starts, and packet detail remains unavailable after expiry. Revised labels and notes are English/French. - -Current acceptance still requires maintainer re-review, especially #167/#169/#174. No upstream merge, stable release or production cutover was performed. The next focused issue is server #183 (saved-route prefix-width changes); broad partial issues remain open. External server #182 and web #93 are unmerged and not in this tested composition. Owners decide the release breakpoint and production switch. - -[Exact current heads, validation and recovery](docs/review-release-20260929.md). - -## Ingest integration delivered — 28 September - -The upstream ingest queue and bounded route-reconfirmation changes are integrated with all pending features. Independent server #166 was updated to preserve advert-name freshness and optional repeat-path delivery together. Suppressed duplicates perform no live-only endpoint work; repeats do not overwrite current names with older adverts. The rest of the queue refreshed without source conflicts. - -All fifteen published application heads pass checks (web CodeQL remains skipped). The composed native Pi server/PostgreSQL suite passes. With 504 scope candidates and blocked route maintenance, an isolated 3,200-input replay preserved the expected 100 packets, 800 observations, 100 decrypted messages and 2,400 opt-in live events with zero queue drops. Native web build/lint and all 935 tests pass. All 18 public assets and both source archives match; English desktop, French phone, scoped packet rows, Public history and packet inspection pass. This is bounded fixture evidence, not proof of universal production losslessness. - -Both real MQTT feeds advanced during the 63-second runtime observation with no restarts or queue-overflow errors. Malformed region and timestamp warnings remain. Valid whole-second timestamps with numeric UTC offsets expose a pre-existing parser gap, now [server #181](https://github.com/MeshCore-Beacon/beacon-server/issues/181). That parser correction is now in #184; see the September 29 record for current priority. The older CoreScope legacy-counter gap remains unexplained. - -September 28 checkpoint: server `2ed2e03117f6c88795d446456e6d74c20c485d28` / web `6b6884951a3dac01b592dfec83f0191879c5696c`. Configuration, schema042, Public key, 504 exact-case candidates, YOW importer and 72h/30d/720h retention are unchanged. A fresh dump restored successfully and was checksum-verified off the Pi. Only Beacon restarted; the other 22 containers were unchanged. Same-schema rollback retains new data and restores server `7c9599b1` / web `dfeb2777` from `ingest-cutover-20260928T222830Z`; frontend-only recovery is `web-20260928T224458Z` paired with the new backend. Exact procedures and current PR heads are in the integration record below. - -The shared UTF-8 corrections and merge guidance remain in place. Both application repos still disable merge commits; the contributor has READ access, so a maintainer must enable that setting. The helper remains optional for maintainers and unrelated to ingestion. Owners retain upstream merges and production release. - -[Current PR heads, validation and recovery](docs/ingest-integration-20260928.md). - -## Earlier Canada/US scope and packet-layout checkpoint — 28 September - -The preview now has **504 exact-case scope candidates**: 261 distinct names from all 237 published Canadian/US regional scope catalogues, plus lowercase IATA/group, province/state, district/territory and country fallbacks. It covers 244 currently known region codes, all 13 Canadian province/territory codes, all 50 US states, DC and five US territories, and includes `#ca`, `#can`, `#us`, `#usa` and `#na`. Counts overlap; do not add these categories together. Published mixed-case names are preserved because case changes the key. Named scopes are not geographic boundaries or evidence of repeater use. - -The reported “Test 4” packet uniquely matches `#ykf`; its original unresolved label remains unchanged. Fresh “Ykf test” and “This is scoped to ykf only” messages now resolve as `#ykf`. A read-only audit of 932 retained transport packets found 670 unique candidate matches, 255 without a known match and seven short-code collisions. Unknown custom names cannot be recovered from these codes alone. Observer/neighbor reports currently contained only the wildcard `*`, so they supplied no additional names. No historical labels were rewritten. The native Pi matcher passes the captured-packet regression and measured a median 0.55ms for a full 504-name scan; this is not a production-capacity claim. - -See the [candidate snapshot and provenance](docs/north-american-scope-candidates.json). This is a recorded catalogue snapshot, with the existing automatic YOW importer still active. For subsequent scope work, refresh Canadian/US published catalogues and observed IATA/report names within API cache/rate limits, retain manual fallbacks and surface unresolved/ambiguous codes. Do not claim an undisclosed recurring all-region discovery service. Arbitrary private names still require a published catalogue or an explicit supplied/reported name. - -[Web #92](https://github.com/MeshCore-Beacon/beacon-web/pull/92), `dfeb2777`, follows #89 and closes #90. It gives the route label and scope separate lines within a 128px track, preserving 37px rows and exact scope case. Long tags remain inside phone cards. A real browser geometry check reproduced the spill before the fix and passed afterward, including the user's `BB2F2752` row. Desktop, 768px table, 390px phone, keyboard and English/French public checks pass. All 935 tests pass on Windows and the Pi, as does exact-head CI; existing warnings remain and web CodeQL is skipped. - -At that earlier checkpoint, the Pi served web `dfeb277756b1a9b230d7e7e0f2d45d62713bf45b` with unchanged server `7c9599b1`, every prior candidate, and the updated [source/changelog](https://canadaverse.org/beacon-dev/source.html). All 18 assets and both source archives match. Both inputs and public LIVE are verified; frontend publication restarted no containers. Application merges/releases remain owner-controlled. The later coordinated refresh above integrates #177–#180 and web #91; this paragraph records the earlier validation. - -## Channel scope investigation and Public channel — 27 September - -[Server #176](https://github.com/MeshCore-Beacon/beacon-server/pull/176) (`7fc631e8`, follows #174, closes #175) and [web #89](https://github.com/MeshCore-Beacon/beacon-web/pull/89) (`e7618fd7`, follows #87, closes #88) expose first-recorded packet scope consistently in history, catch-up and live messages. The interface adds scope filtering, packet inspection, distinct unknown/unavailable/unscoped states and English/French explanations. Duplicate broker messages and live arrivals during a history request preserve the existing page cursor and message counts. Imported catalogue names alone are not forwarding evidence. - -At the channel-scope checkpoint, the Pi preview was composed server `7c9599b167bcad2b416ef03304ff27909ab3021b` / web `e7618fd7d40aff4e01f581999fdbb21d10de2e67`, with every earlier review candidate included. Native server/PostgreSQL tests, all 935 web tests, build/lint and published-head CI pass (web CodeQL is skipped). Browser checks cover live/history filtering, keyboard inspection, English/French and a 390px phone. The public page is LIVE, both MQTT inputs are connected, and all 18 assets plus both source archives match the [changelog/source offer](https://canadaverse.org/beacon-dev/source.html). - -The standard MeshCore Public channel key is enabled at the user's request, matching the known non-hashtag hash-11 channel on dev.meshcore.ca. A restored-copy trial recovered 2,735 retained messages in 16.127 seconds; public decoded history and a new incoming message were verified. Expired packets remain unavailable. Schema042 is unchanged. Rollback retains server `eb99f752`, web `98f820d2`, configuration and source without discarding new database rows; the older schema041 recovery remains separately available. Packets stay 72h, summaries 30d and telemetry 720h. Public admin, backups and foreign detection remain disabled. - -See the [boundary integration plan](docs/meshmapper-boundaries-plan.md). - -**Next:** refresh review feedback and listed issues first, then optional MeshMapper boundary synchronization using the published Zones API and existing map layer. Preserve manual boundaries and cached geometry; keep cross-boundary packet/route investigation in a later focused contribution. Group scope catalogues, regional evidence, existing translations and broader parity work remain open. Maintainers control acceptance, merge order and production release. - -## Regional scope import — 27 September - -[Server #174](https://github.com/MeshCore-Beacon/beacon-server/pull/174), `71e4e871`, follows #172 and closes focused issue #173 on acceptance. It imports public MeshMapper catalogue names into the existing matcher, keeps manual settings, persists last-known-good data and retry timing, and reports synchronization status in operator logs. Regional candidate limits and short-code ambiguity checks prevent a catalogue entry from becoming a forwarding claim. The feature defaults off; channel tags remain a separate slice. - -At the scope-import checkpoint, the Pi ran composed server `eb99f7523727b7e4208667e201f50ad235ce2c5f` with unchanged web `98f820d2`. Every prior review candidate remains included. Its explicit Ottawa/YOW source imported seven names; the public filter and actual incoming `#yow` adverts were verified. Native PostgreSQL tests and published-head CI/race/security checks pass. The maximum-catalogue matcher fixture took about 0.15ms per transport packet on the Pi; this is a microbenchmark, not a production-capacity claim. Pi ThreadSanitizer cannot run with the host's address layout, so race validation is on Linux CI. - -Migration 042 preserved all 31 existing table fingerprints in a restored copy. Immediate rollback retains original schema041 database `beacon_pre042_20260927` and backup `scopes-cutover-20260927T221330Z`; a checksum-verified private dump is also off the Pi. Only Beacon restarted, 22 other containers were unchanged, and both feeds reconnected. The [changelog and source](https://canadaverse.org/beacon-dev/source.html), 18 web assets and English/French-mode desktop/phone scope filtering were checked. Existing untranslated packet controls remain under #12; web source is unchanged. Retention remains 72h/30d/720h, and public admin/backup/foreign detection stay disabled. - -**Follow-up delivered above:** channel tags are in #176/#89. Group catalogues and regional evidence remain separate future slices. Maintainers retain merges, stable releases and production cutover. - -## Observer investigation navigation — 27 September - -[Web #87](https://github.com/MeshCore-Beacon/beacon-web/pull/87), final candidate `98f820d2b2af5a69faf3f9aaf30d5f14b45feea8`, follows #85 and closes focused issue #86 on acceptance. Escape/Close dismisses the active panel and restores focus. Observer adverts are keyboard buttons and select the exact packet observation; embedded packet inspection preserves its originating URL. Revisiting an open entity returns to its existing panel, while tab/region/entity changes discard obsolete panels. - -Opening an observer dashboard keeps one originating screen mounted under its original Router location. Period, picker, comparison and directory detours stay within that visit; Back or the translated return action restores route/packet/node/map/Analytics state. A real route retained its typed filter, sort and 720px scroll position; a panned map's copied centre/zoom/layer/node link was identical after return. The Analytics leaderboard uses the same handler and provides keyboard buttons beside the canvas, using existing data. Direct/copied/reloaded dashboards remain standalone; arbitrary in-memory state is not persisted across reload. - -At the observer-navigation checkpoint, the Pi served web `98f820d2b2af5a69faf3f9aaf30d5f14b45feea8` on server `99e623c5`. Final native build/lint and **929 tests in 106 files** pass; exact-head CI passes (web CodeQL remains skipped). All 18 public assets and both source archives match. Public LIVE, both MQTT feeds and return/keyboard journeys pass. A real packet-list regression is fixed: the retained origin stays invisible/inert with its layout intact. Final public checks preserve 442px scroll and 506px viewport height throughout the visit, plus the selected report URL. No containers restarted. Immediate web rollback is `5a261162` at `web-20260927T210108Z`; the phase-start `3b3abdcc` recovery remains at `web-20260927T202233Z`. Existing database recovery and 72h/30d/720h retention policies are unchanged. All prior review candidates remain included. - -**Observer follow-up:** web #12 still covers the observer directory and quick-detail translations. Node/trace presentation and further reach/timing analytics follow the approved roadmap. Scope import #174 and channel tags #176/#89 are implemented above. Maintainers control merges, stable releases and production cutover. - -## MeshMapper scopes contract — 27 September - -The [public API](https://wiki.meshmapper.net/scopes-api/) is available. The documented YOW endpoint returned HTTP 200 and a successful conditional HTTP 304; it requires no API key. The [scope integration plan](docs/mesh-scopes-plan.md) now specifies explicit per-IATA sources, cached refresh, durable last-known-good data, manual-name preservation and separate imported/observed evidence. Group results cannot be attributed to individual member IATAs. The former unpublished-endpoint blocker is removed; importer #174 and channel tags #176/#89 are now deployed for review. Review feedback and listed issues retain priority. That contract-only update preceded the tested importer deployment recorded above. - -## Saved-route evidence — 27 September - -[Server #172](https://github.com/MeshCore-Beacon/beacon-server/pull/172) (`f473b165`) and [web #85](https://github.com/MeshCore-Beacon/beacon-web/pull/85) (`3b3abdcc`) implement the next connected-investigation slice. They close focused server #171 / web #84 on acceptance. A full saved route now links to paged retained reports, exact packet inspection, reporting observers and named hops. Existing packet inspection leads to the selected report's map. Shared route links pin the effective server time window; browser Back and in-place inspection preserve the route/filter context. - -Matching uses the complete saved prefix bytes, width and IATA, not confirmed physical identity. Other widths, subsegments, TRACE and unclassified records are excluded. Retained reports are separate from distinct-packet or historical route totals. Empty/expired evidence and unavailable prefixes are explained. Requests default to 24 hours, permit at most 30 days and fetch 50 reports per page; the interface caps each investigation at 500. New interface text is English/French, with a compact phone layout and expandable counting definitions. - -The stack was refreshed once for accepted server #170 (`dec643a2`): optional exact WebSocket origins and opt-in observer public-key fields, not authentication. Updated server order: #167 (`02743704`) -> #169 (`91b21995`) -> #172 (`f473b165`); independent #166 (`2c5ad5fc`) remains included. Web order: #75 -> #79 -> #80 -> #81 -> #83 -> #85. All candidates remain reviewable separately; do not rebase every child independently. Deploy the server API before its web consumer. - -At the saved-route checkpoint, the preview backend was composed `99e623c56477fd9b667d5f56bfb1a0eff34ecb2b`. Its complete native Pi/PostgreSQL suite passed. Restoring a fresh private dump and adding migration 041 preserved all 31 table fingerprints. The live switch retained the original schema040 database as `beacon_pre041_20260927`; rollback directory is `route-cutover-20260927T190644Z`. Both MQTT feeds reconnected and 22 unrelated containers were unchanged. Packets remain 72h, archived summaries 30d, telemetry 720h. Public admin, backup and foreign detection remain disabled. - -At the route-evidence checkpoint, the Pi served web `3b3abdcc5bfa85b60c8959715736ea42c3d0bbd8`. The final native build/lint and all **915 tests** pass; exact-head CI passes (web CodeQL remains skipped). Desktop and 390px phone, English/French, copied/shared links, Back, keyboard Close/focus, exact report/node/observer and selected-map journeys were verified. All 18 public assets and both source archives match the tested artifacts. The public page is LIVE; both MQTT feeds are connected. Frontend publication restarted no containers. Web rollback retains `1d5d65e` in `web-20260927T192154Z`; the [source/changelog](https://canadaverse.org/beacon-dev/source.html) lists the complete review composition. - -**Follow-up implemented above:** [web #86](https://github.com/MeshCore-Beacon/beacon-web/issues/86): observer quick-inspection Escape handling and broader cross-tab/overlay return navigation. That earlier Escape limitation is addressed by #87; acceptance remains with the maintainer. Node/trace evidence and additional reach/timing analytics follow that connection work. MeshMapper scope import #174 is now implemented and recorded above; its later channel/UI slices remain open. Broad server #60/#72/#99/#116 and web #12 remain open; this phase does not claim full parity or production capacity. Maintainers retain merges, stable releases and production cutover. - -## Packet reception investigation — 27 September - -[Web PR #83](https://github.com/MeshCore-Beacon/beacon-web/pull/83), `1d5d65e2807c2cb53a998743212d9a5b64ba80c4`, follows #81 and closes focused issue #82 when accepted. It adds grouped retained packet reports, selected-report links, observer inspection/dashboard access and a selected-path map. The initial list stays compact and keeps the selected group open. Equal prefixes are not treated as confirmed identical physical routes; empty/missing paths and TRACE intended routes have explicit labels. - -Map projection omits ambiguous/unlocated identities and breaks lines at gaps. Live animations across uncertain chains are suppressed, so fewer speculative lines appear. Unavailable selected paths no longer silently show All paths. A shared-path loading race is fixed by checking the requested packet hash. Packet labels use the existing Noto Sans stack; external basemap emoji-glyph/sprite fallback warnings can still occur. - -At the packet-investigation checkpoint, the Pi ran web `1d5d65e` with server `88c2c10c`. Native build/lint and all **906 tests** pass; focused Windows checks and desktop/390px phone/English/French/keyboard/Back/shared-link checks pass. Public assets and source match, both MQTT feeds are connected, and the frontend publication restarted no services. Earlier review candidates remain included. The [changelog/source](https://canadaverse.org/beacon-dev/source.html) identifies the running build. Maintainers still own merges, stable releases and production cutover. - -**Follow-up:** the saved-route evidence phase above implements this API and interface. Remaining work is observer return navigation. Non-packet overlay return navigation remains a separate follow-up. The separate MeshMapper scope plan now has importer #174 and channel tags #176/#89 implemented. Broader server #60/#72/#99/#116 and web #12 remain open; this is a first connected-investigation slice, not full parity. - -## Direction - -Bring useful CoreScope investigation and analytics features into Beacon's existing ingest, database, API, cache and web components. Prioritize review regressions and measured stability/performance problems, then useful analytics pages. Keep each API or page a focused contribution with explicit counting semantics and validation. - -Current sites: - -- [Beacon reference deployment](https://dev.meshcore.ca/) -- [CoreScope reference deployment](https://live.meshcore.ca/) -- [Development preview](https://canadaverse.org/beacon-dev/) and its [changelog/source](https://canadaverse.org/beacon-dev/source.html) - -| Repository | Responsibility | Contribution target | -|---|---|---| -| [beacon-server](https://github.com/MeshCore-Beacon/beacon-server) | Ingest, storage, read models and public/protected APIs | `dev` | -| [beacon-web](https://github.com/MeshCore-Beacon/beacon-web) | Investigation tools and analytics pages | `dev` | -| [beacon-docs](https://github.com/MeshCore-Beacon/beacon-docs) | Shared contracts, operator guidance and this roadmap | `main` | -| [beacon-mobile](https://github.com/MeshCore-Beacon/beacon-mobile) | Mobile client; coordinate API compatibility | `main` | - -## Observer-first release delivered for review - -The observer-first release is implemented in four focused review candidates: [server #169](https://github.com/MeshCore-Beacon/beacon-server/pull/169) (`91b21995`, closes #168), [web #79](https://github.com/MeshCore-Beacon/beacon-web/pull/79) (`3a0eb4c8`, closes #76), [web #80](https://github.com/MeshCore-Beacon/beacon-web/pull/80) (`b3a6b088`, closes #77) and [web #81](https://github.com/MeshCore-Beacon/beacon-web/pull/81) (`42ae09b7`, closes #78). All are out of draft. Server #169 follows #167; web order is #75 -> #79 -> #80 -> #81. Independent server #166 remains in the preview composition. Maintainers control acceptance and release; these issues remain open until their changes are accepted. - -At the observer-release checkpoint the Pi ran composed server `88c2c10c830034cee70a46fca717af518803a544` and web `42ae09b7c6be50cd0f617bad8aea35825be9f12e`, with the [updated changelog and exact source](https://canadaverse.org/beacon-dev/source.html). Raw packets remain 72 hours, archived hourly analytics 30 days, telemetry 720 hours. All four candidates passed their native Pi suites; the final web build passes 895 tests. Server tests use actual PostgreSQL. Windows server/full destination/dashboard validation and 48 focused final comparison/API tests also pass. Desktop, 390-pixel phone, English/French, keyboard, legacy/shared links and Back/search/sort/scroll were checked. Physical iPhone Safari and production-volume capacity remain separate gates. - -Migration 040 preserves original rows and repairs available archived unknown-payload counts without inventing signal samples. A restored clone passed the migration probe. A fresh private dump was copied off the Pi and its checksum verified; the original schema039 database and matching binary/config/source are retained for DB-aware rollback. Only the Beacon app restarted for the server change; 22 other containers were unchanged. Frontend publication restarted no services. Both MQTT feeds reconnected. Public admin, backup and foreign detection remain disabled. - -See the [observer implementation and subsequent UX releases](docs/observer-monitoring-plan.md) and the separate [Mesh Scopes interoperability plan](docs/mesh-scopes-plan.md). - -## Accepted consolidation batch - -All ten original server/web contributions merged on September 24. The September 25 web batch is also accepted: #70 contains translations #63/#64/#65/#66/#69 plus Timestamp wording; #68 and #72 merged separately and closed #67/#71. The five earlier translation PRs were closed as included, with exact ancestry/tree equivalence verified. Web #73 and server #162/#163 then landed. That acceptance batch cleared the application queue. The new September 26 retention/endpoint PRs and docs #5 are now in review, as listed below. - -At the September 27 consolidation checkpoint, accepted dev was server `dec643a2ade712cd60feb2bc761a5654c7c82714` / web `54b5093ac0302c7db9d51e1d7fe23570eae7cdb5`. Exact-head CI/image builds pass; server coverage/CodeQL pass and web CodeQL remains skipped. Stable releases are still server **v1.6.0** and web **v1.3.0**. The original September 24 freeze (`c02317a4` / `0f0a6ca5`) remains historical evidence, not proof that the newer server migrations are Pi-validated. - -| Accepted PR | Scope | Merge commit | -|---|---|---| -| [Server #149](https://github.com/MeshCore-Beacon/beacon-server/pull/149) | add protected account lifecycle endpoints | `4d2642ab` | -| [Server #154](https://github.com/MeshCore-Beacon/beacon-server/pull/154) | add protected database and config download | `a25e2675` | -| [Server #157](https://github.com/MeshCore-Beacon/beacon-server/pull/157) | add bounded reception signal analytics | `6107578c` | -| [Server #159](https://github.com/MeshCore-Beacon/beacon-server/pull/159) | add bounded path and hash-width analytics | `8a3fa7e6` | -| [Server #160](https://github.com/MeshCore-Beacon/beacon-server/pull/160) | verify native archives offline without extraction | `20691dc5` | -| [Server #161](https://github.com/MeshCore-Beacon/beacon-server/pull/161) | summarize ACK and trace references | `c02317a4` | -| [Web #55](https://github.com/MeshCore-Beacon/beacon-web/pull/55) | add RF and signal analytics | `2405e1ac` | -| [Web #57](https://github.com/MeshCore-Beacon/beacon-web/pull/57) | add Paths and Hashes analytics | `929ba83c` | -| [Web #59](https://github.com/MeshCore-Beacon/beacon-web/pull/59) | distinguish analytics navigation icons | `c2ab29a5` | -| [Web #61](https://github.com/MeshCore-Beacon/beacon-web/pull/61) | omit reset and invalid node locations | `0f0a6ca5` | - -Server #156/#158 and web #54/#56/#58/#60 are closed. Server #60/#72/#99/#116 and web #12 remain open for their remaining scope. Web #67/#71 are closed following #68/#72. Archive verification establishes structure and integrity, not authenticity, SQL safety or restorability. Packet-carried ACK/TRACE/PING references are not identity or delivery guarantees. - -## September 26 retention and endpoint fixes - -The Pi was first matched to accepted dev server `91b4b457` / web `54b5093a`, including native validation and a verified restore across migrations 037/038. At that checkpoint it ran composed server `a8394f10` and web `3a18e6d1`, adding three focused review candidates: - -| PR | Head | Scope / closure | -|---|---|---| -| [Server #166](https://github.com/MeshCore-Beacon/beacon-server/pull/166) | `74f16de8` | First/renamed adverts resolve after the node update; closes #164 | -| [Server #167](https://github.com/MeshCore-Beacon/beacon-server/pull/167) | `76428b1b` | 30-day hourly summaries survive raw packet expiry; closes #165 | -| [Web #75](https://github.com/MeshCore-Beacon/beacon-web/pull/75) | `3a18e6d1` | Additional-match count and all endpoint candidates on hover, keyboard or touch; closes #74 | - -All are out of draft. Exact-head build CI passes, server CodeQL passes, and web CodeQL remains skipped. MrAlders0n/Claude review was requested in PR comments because formal review requests are unavailable to the contributor account. Both server PRs are independent on the same accepted dev and may merge in either order. Web #75 is also independent. The workflow records #167 plus #166 as a complete PR/head preview input; its separate manifest can be refreshed after upstream changes. No routine manual restacking is required for these non-overlapping changes. No upstream PR was merged by the contributor. - -The agreed Pi policy is **72-hour raw packets, 30-day hourly analytics and 720-hour telemetry**. Migration 039 archives compact summaries as each raw packet cohort expires, atomically with deletion, without storing bodies or raw paths. Traffic, payload, top observers, talkers, advertisers, observer activity, Signal and Paths use the retained summaries. Already-purged history cannot be recovered. Packet detail, sub-hour activity and exact observer comparisons still use retained raw data; entity/scope/radio population counts keep their current meaning. - -Full Windows and native Pi Go/PostgreSQL checks pass, including rollback on archive failure, retries, concurrent ingestion, late observations, distinct observers across batches, multiple IATAs, nullable/radio/path semantics and independent 30-day expiry. The full frontend build/lint and **874 tests** pass. A 1,001-packet fixture with 1KB bodies compacted to at most twelve archive rows; the first Pi run took 85ms for archive/deletion, which is a fixture measurement rather than a production-throughput guarantee. A restored copy of the actual preview database retained all eight view counts after every raw packet was deleted in a rolled-back test. Source/index hashes and the public candidate popup were verified. - -Migration 039 preserved fingerprints of all 23 original application tables. The actual prior schema038 database, exact binary/configuration and private dump remain available for rollback; older pre-038 recovery is retained separately. Only the Beacon preview app restarted (about 33 seconds); 22 other containers were unchanged and both MQTT feeds reconnected. Public admin/backup and foreign detection remain disabled. [Current changelog and corresponding source](https://canadaverse.org/beacon-dev/source.html). - -Current broader issues remain server #60 (admin), #72 (backup/import), #99 (packet summaries), #116 (MQTT investigation), and web #12 (remaining translations). Current priorities are review feedback and server #181, followed by boundaries and the remaining investigation work. A measured month of accumulated history, production-scale capacity and physical Safari checks remain separate gates. Maintainers own stable releases and the owners handle production cutover. - -## Delivered foundations - -- Faster bounded node, route, trace and clock-stat queries; list-limit validation and NULL-observation handling. Representative changes: [#111](https://github.com/MeshCore-Beacon/beacon-server/pull/111), [#118](https://github.com/MeshCore-Beacon/beacon-server/pull/118), [#120](https://github.com/MeshCore-Beacon/beacon-server/pull/120), [#122](https://github.com/MeshCore-Beacon/beacon-server/pull/122), [#124](https://github.com/MeshCore-Beacon/beacon-server/pull/124). -- MQTT client isolation, proxy identity handling, API/WebSocket limits and interrupted-index recovery. Timeout attribution in #116 remains separate from these accepted fixes. -- Observer age-out, advert summaries, batched endpoint resolution and companion matching (snapshots superseded by #163), stable channel paging, packet search and shared packet links. -- Observer activity/telemetry and comparison, regional scope statistics, runtime administration, optional foreign-repeater detection and the backup-export foundation. - -## Analytics delivery and counting rules - -| Page | Current behavior | Important interpretation | -|---|---|---| -| [Traffic](https://canadaverse.org/beacon-dev/?tab=Analytics&statsTab=traffic&range=24h) | UTC hourly heatmap, IATA trends, reception share and exact counts | Counts reported receptions; missing hourly records remain gaps | -| [Scopes](https://canadaverse.org/beacon-dev/?tab=Analytics&statsTab=scopes) | Regional packet, observer-membership and default-scope-node charts | Retained counts have no rolling date filter; memberships can overlap | -| [RF / Signal](https://canadaverse.org/beacon-dev/?tab=Analytics&statsTab=signal&range=24h) | SNR/RSSI distributions, hourly means, sample coverage and exact tables | Missing/non-finite and unavailable zero/zero readings are excluded per metric; a measured zero SNR remains valid | -| [Paths & Hashes](https://canadaverse.org/beacon-dev/?tab=Analytics&statsTab=paths&range=24h) | Hash-width share, received header-entry distribution, hourly trends and coverage | Empty paths never vote for width; TRACE paths hold signal readings; unusable metadata is unclassified | - -The path page counts stored receptions, not unique devices. Flood paths accumulate entries, while direct routes carry remaining entries. Observed widths do not establish device capability or collision rates. Signal readings describe reception at the reporting observer rather than a complete end-to-end link. - -The September 20 review correction moves both new aggregate APIs onto materialized hourly snapshots, preserving reception/region semantics and normalizing polling windows to UTC hours. In a rolled-back million-row Pi fixture, Signal request queries took 1.8–77.5 ms and Paths 3.5–141.9 ms across custom/generic plans. Initial population took 14.4/8.4 seconds, refresh 16.8/6.2 seconds, and view/index storage was 6.5/11.3 MB respectively. These measurements cover the fixture, not the full production workload. See the [release consolidation checklist](RELEASE-CHECKLIST.md) for remaining gates. - -September 20 validation covered native Go/PostgreSQL/HTTP behavior and **786 web tests**, plus private backup compatibility, feature-only startup failure, TLS/password files, cancellation and schema/data/sequence restoration. That review update passed 390/1280px browser checks, with ten distinct glyphs and complete-hour text; earlier chart checks also covered 320/768px. This is historical evidence for the unchanged feature code. Current revisions and corresponding-source archives are on the preview's changelog page. - -The September 24 consolidation check built accepted server `c02317a4` and retained web `42ba5fcb`, identical in source to accepted `0f0a6ca5`. Its native PostgreSQL and public/browser evidence remains in the release checklist. The Pi frontend contains the now-accepted translations and #68/#72 as documented above; the prior combined preview is retained for rollback. The three-hour retained analytics sample does not establish 7/30-day history or production capacity. - -## Next phases - -Use the [active post-1.4 sequence](docs/post-140-roadmap.md). The -older dependency queue and optional-boundary implementation are delivered in the -current candidate and must not be scheduled again. The next focused implementation -is server #183; Atlas follows separately after 1.4.0. Scope/route crossing analysis, -node/trace presentation and new analytics retain their own acceptance contracts. - -## Listed work still open - -| Issue | Remaining scope | -|---|---| -| [Server #183](https://github.com/MeshCore-Beacon/beacon-server/issues/183) | Active first post-1.4 phase: saved-route metadata and evidence consistency across width changes | -| [Web #94](https://github.com/MeshCore-Beacon/beacon-web/issues/94) | The 24h/3d correction landed through #95/#99, then upstream restored 7d/30d; the retention-window decision remains open | -| [Server #116](https://github.com/MeshCore-Beacon/beacon-server/issues/116) | #179/#180 are integrated and pass bounded replay/route-lock checks; attributing the historical incident still requires matching evidence | -| [Server #99](https://github.com/MeshCore-Beacon/beacon-server/issues/99) | Advert names and ACK/TRACE/PING references are accepted; define any remaining packet-type formats | -| [Server #60](https://github.com/MeshCore-Beacon/beacon-server/issues/60) | Remaining administration/worker/persistence behavior; account records do not establish login sessions | -| [Server #72](https://github.com/MeshCore-Beacon/beacon-server/issues/72) | Download #154 and archive validation #160 are accepted; import, browser access, deployment-file coverage and remote/scheduled backup remain | -| [Web #12](https://github.com/MeshCore-Beacon/beacon-web/issues/12) | Foundation and translations through #70 are accepted; Mesh/Talkers fixes are merged. Remaining analytics/screens/dialogs/formatting still need translation | - -The six completed analytics/icon/map issues are closed. Refresh all currently open issues and PR feedback first at every continuation; accepted partial contributions are not grounds to close broader issues. Use closing references only when a PR completes the issue's accepted scope; use related references for partial work. - -## Production parity matrix - -Matching tab names is not acceptance. Each capability needs verified semantics, time/region behavior, empty/partial data, performance and browser evidence. - -| Capability | Coverage / remaining evidence | -|---|---| -| Overview | Mesh overview and Traffic; reconcile packet/reception grain and retained windows | -| RF / Signal | New distributions, weighted means and sample coverage; production-volume measurements remain | -| Topology / route patterns | Existing routes, traces and neighbour graph; deeper edge/subpath/connectivity analysis remains | -| Channels | Directory/chat/talkers exist; traffic statistics, unknown-channel and history behavior remain | -| Hash statistics | Paths & Hashes covers observed ordinary widths and entries; trace-payload widths are distinct | -| Hash issues | Endpoint/path ambiguity primitives exist; observed ambiguity and static conflicts need separate views | -| Node analytics | Existing directory/detail/observations; richer attributed activity, signal, payload and peer analysis remains | -| My Repeaters | Owner selection/watchlist behavior and grouped analytics need an agreed contract | -| Repeater metrics | Observer telemetry overlaps partially; units, resets and role attribution need reconciliation | -| Distance | Coordinates/maps exist; valid link/path distances, unknown positions and confidence remain | -| Neighbour graph | Existing graph needs retained scaling, filter and accessible-fallback acceptance evidence | -| RF health | Noise/airtime/error telemetry exists; comparable health views need real samples and valid deltas | -| Clock health | Existing clock-drift endpoint/UI; retain role, threshold and history semantics | -| Roles | Node-type census exists; activity and unknown-role interpretation need acceptance evidence | -| Scopes | Regional API and new page exist; reconcile against populated retained data | -| Prefix tool | Public-prefix inspection/simulation remains unverified | -| Observer monitoring/comparison | Unified dashboard and contextual comparison in #169/#79/#80/#81; see the current release above. Shared windows and deduplication remain explicit; raw overlap can expire earlier than summaries | -| Other workflows | Validate decoder/search/sharing, live map/replay, settings, optional clients and legacy links | - -## Release gates - -- Inventory actual CoreScope retention, earliest/latest durable data, configuration and recovery copies. Example retention settings and public in-memory counts are not production history evidence. -- Reconcile Beacon's deduplication, identity, encryption/key and time semantics. Document whether migration or sufficient parallel ingestion supplies each historical window. -- Complete backup recovery scope with private disposable restore tests, including schema/data/sequence continuation and the deliberately excluded deployment files. -- Measure cold start, reconnects, ingest freshness, CPU/memory/storage, query latency and maintenance against representative production volume. Long-window raw aggregates may need rollups. -- Validate keyboard/mobile/browser behavior, chart readability and sharing. Physical iPhone Safari and a sustained load/connection soak remain open validation gaps. -- Verify release artifacts from reviewed source and applicable CI. The upstream web CodeQL workflow is currently disabled; its skipped job does not count as a security scan. -- Publish matching source, configuration guidance, known limitations and a verified rollback procedure. The deployment owner performs the production switch. - -The development preview still has limited accumulated history; configured 30-day retention does not mean a measured month is available. MeshMapper catalogue import is enabled only for the explicitly configured YOW preview source. Its public admin/backup and foreign detection are disabled. These limitations remain explicit until configuration and validation support enabling them. - -## Keeping this roadmap useful - -Update this file when a phase is delivered, a dependency merges, an issue closes or the next priority changes. Keep private configuration and host-specific operational records outside this repository. Link current GitHub work and the public source/changelog so another contributor can continue without a private workstation path. +# Beacon primary roadmap: 2.1–2.2 + +**Primary planning authority — adopted 4 October 2026.** This is the roadmap +Torchlight and the Canadaverse preview work should use going forward. It records +the operator-supplied planning discussion and MrAlderson's follow-up. It supersedes +the older parity roadmap's release priorities, not its historical evidence. + +These are agreed goals, **not a declaration that the features are implemented, +hardware-qualified, accepted upstream or deployed**. Track completion against +reviewable code, tests and the actual environment. Keep the 2.1 scope focused. + +## 2.1.0 — Foundations + +- Add the basic **Atlas and Topology pages**, using the existing preview as the + starting point rather than introducing a competing workflow. +- Include an initial **Beacon agent** implementation/scaffold. Its precise + component responsibilities and acceptance criteria need to be written down + before implementation; “initial” does not mean a fully completed agent. +- Have the **MQTT layout and setup working**: document the topics, message + contracts, identity/authorization boundary and setup flow, then demonstrate an + end-to-end accepted sample. Do not confuse MQTT intake permissions with RF + polling permission. +- MrAlderson will help with Atlas. Keep ownership and remaining work visible + without assuming a contributor has completed or accepted an unreviewed change. +- Preserve the current Topology approach as the baseline; gather specific feedback + on appearance and interaction instead of expanding this release into a redesign. + +## 2.1.1–2.1.9 — Telemetry and collection in Atlas + +- Expand telemetry collection and its presentation within Atlas incrementally. +- The **mobile application is an opt-in collector**, connecting to a companion + radio over BLE and running in the phone background where the operating system + permits. Background/BLE reliability must be qualified, not assumed. +- Provide the alternative **Windows/Linux USB-serial collector**, running in the + tray/background after a simple one-time setup. +- Let users select local repeaters from the companion's contact list and supply + guest/admin credentials locally when required. Never publish those credentials + in dashboards, issues, source repositories or telemetry payloads. +- Use the same conservative polling contract on both clients: no more than one + polling attempt per target per hour, a three-hop eligibility limit, initial flood + discovery, reuse of learned/saved routes and a rate-limited flood fallback. +- Send collected readings through the agreed API/intake contract. Beacon should + show verified, attributed samples on each node's Atlas card or its linked node + view, with freshness, units, sensor channel and visible gaps. +- Coordinate the mobile client, desktop collector, Atlas and server API as one + end-to-end feature, retaining their separate responsibility boundaries. + +## 2.2 — Complete Atlas and refine Topology + +- Complete the agreed Atlas experience, including accepted telemetry and + collection integration. +- Refine Topology using observed usability feedback and the current implementation. +- Fix major and minor defects and address measured performance problems across + the accepted system. Broader feature ideas remain separately prioritized backlog. + +## Collector safety and acceptance gates + +The one-hour minimum is a maximum polling rate, not an instruction to transmit +every hour regardless of congestion. Longer intervals and deferred attempts remain +valid. Restarts, reconnects, failures, parallel collectors and manual refreshes must +not bypass the budget. Login, retries, discovery and route fallback must be counted +and bounded; uploading an already-collected sample is not a new RF poll. + +The **three-hop requirement is agreed**, but a known route length does not by +itself cap flood propagation. Before promising a hard RF reach limit, verify what +the companion firmware and API can enforce. If it cannot be enforced, defer that +poll and report the limitation; do not silently change the radio configuration or +describe an unrestricted flood as “three hops.” + +“Verified” means the intake contract's authenticated/validated provenance and +accepted sample checks. It must not imply hardware attestation or guaranteed sensor +truth unless those capabilities are implemented and demonstrated. + +Use persistent per-target scheduling, conservative shared-radio budgets, bounded +backoff and congestion checks. Cross-client duplicate-poll protection remains a +wider-rollout qualification gate. No remote reboot, arbitrary radio CLI or general +administrative control is implied by this roadmap. + +## Development and delivery + +Prepare reviewable work against the agreed upstream `dev` base and keep the +Canadaverse forks/preview independently verifiable. MrAlderson requested a branch +handoff and a `dev1` image/workflow that the development host can pull. Record its +exact branch, image/tag contract and maintainer approval before changing that host. +This document does not itself enable a workflow, merge upstream or deploy to +`live.meshcore.ca` or `dev.meshcore.ca`. + +Torchlight may carry out authorized preview-fork development under its existing +standing authority. It should use this roadmap to prioritize and maintain task +states, checkpoints, issues/PRs, source revisions and preview verification. Future +ideas do not automatically become 2.1 requirements. Escalate scope changes instead +of silently adding them to a release. + +## Supporting evidence and history + +- [Current preview](https://canadaverse.org/beacon-dev/) and + [published build metadata](https://canadaverse.org/beacon-dev/build.json). +- [Torchlight project dashboard](https://canadaverse.org/torchlight/). +- [Collector research and earlier design discussion](docs/beacon-21-collector-design.md): + useful implementation evidence; conflicting earlier priorities yield to this roadmap. +- [Broader parity backlog](docs/post-140-roadmap.md): supporting backlog, not the + default release sequence. +- [Previous roadmap and dated checkpoints](https://github.com/n30nex/beacon-docs-contributions/blob/483e71188fb000eeb33db080936573c06e96f8a2/ROADMAP.md) + remain preserved in Git history. Old deployment, version and rollback statements + are historical and must not be applied as current operating instructions. diff --git a/docs/beacon-21-collector-design.md b/docs/beacon-21-collector-design.md index ecb7392..be28391 100644 --- a/docs/beacon-21-collector-design.md +++ b/docs/beacon-21-collector-design.md @@ -1,5 +1,15 @@ # Beacon 2.1: Atlas and Collector planning scratchpad +> **Primary roadmap:** [Beacon 2.1–2.2](../ROADMAP.md), adopted 4 October 2026, +> controls priorities and agreed requirements. It includes Atlas/Topology plus an +> initial Beacon agent and working MQTT setup in 2.1.0; telemetry/collection +> expansion in 2.1.1–2.1.9; and complete Atlas, Topology refinement, bug fixes and +> performance in 2.2. Mobile BLE and Windows/Linux USB collectors share the hourly +> maximum, three-hop requirement and flood/learned-route/fallback policy. The +> hardware enforcement and qualification caveats below remain important. Earlier +> “proposed,” USB-first, open-hop-threshold and provisional-release statements in +> this dated scratchpad are historical, not competing primary instructions. + 3 October 2026. This is a source-backed planning note, not a deployment instruction or an accepted upstream release contract. The supplied discussion proposes Atlas and Collector as the 2.1 focus, USB/Wi-Fi first, and MQTT forwarding. Maintainer From 8a7cc7b90b23a447726823e3f5b9771d8b7b2e71 Mon Sep 17 00:00:00 2001 From: n30nex Date: Sun, 4 Oct 2026 15:20:42 -0400 Subject: [PATCH 64/69] [Torchlight] docs: retain future firmware telemetry interoperability [skip ci] Agent: Torchlight --- ROADMAP.md | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/ROADMAP.md b/ROADMAP.md index 27fc17b..b15c72d 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -53,6 +53,40 @@ reviewable code, tests and the actual environment. Keep the 2.1 scope focused. - Fix major and minor defects and address measured performance problems across the accepted system. Broader feature ideas remain separately prioritized backlog. +## Future interoperability — firmware-direct telemetry (FW01) + +**🟢 Keep the direction; 🟡 design and integration remain. No release assigned.** +The [owner's 4 October clarification](https://discord.com/channels/1507764602253869197/1507788454774182030/1556384736522281032) +keeps collaboration with repeater and observer firmware projects open, including +projects like [MeshCore Observer](https://observer.gessaman.com/). + +Wi-Fi-capable devices should be able to opt in and configure a Beacon deployment's +telemetry collection endpoint, submitting their own readings through the shared, +versioned API/intake contract. This complements mobile/USB collectors rather than +requiring an intermediary phone or computer for a device's own telemetry. Keep the +producer contract reusable across clients; do not tie it to one firmware vendor. + +Before claiming support, agree and qualify: + +- Endpoint selection, explicit enable/disable and scoped enrollment, authentication + and revocation. Wi-Fi and node administration credentials stay local, never in + telemetry payloads; intake permissions do not grant remote radio administration. +- Device/producer identity and provenance, sample timestamps, units and optional + sensor fields. Missing/null values must not become fabricated zero readings; + authenticated provenance is not hardware attestation or guaranteed sensor truth. +- Bounded upload cadence, retries/backoff, offline buffering and duplicate handling, + with honest freshness/gaps in Atlas/node cards. Transport/schema details and + partner compatibility remain design work; no endpoint shape is promised here. +- Coexistence with existing observer MQTT packet/status ingestion. Uploading a + device's own readings over Wi-Fi is not an RF poll. If firmware also polls peers, + the existing polling budgets, hop constraints and qualification gates still apply. + +The reference project's public setup page documents Wi-Fi and MQTT configuration; +that is not verification of native Beacon telemetry-API support or a collaboration +agreement. No partner outreach or firmware qualification has occurred in this update. +This future direction does not add a new 2.1/2.2 delivery requirement, authorize +firmware changes or imply remote console/control support. + ## Collector safety and acceptance gates The one-hour minimum is a maximum polling rate, not an instruction to transmit From 88ac4167eea13b151f402f230887215e312046b1 Mon Sep 17 00:00:00 2001 From: n30nex Date: Sun, 4 Oct 2026 16:24:31 -0400 Subject: [PATCH 65/69] [Torchlight] docs: assign backlog to version roadmap targets [skip ci] Agent: Torchlight --- ROADMAP.md | 150 +++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 134 insertions(+), 16 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index b15c72d..39ea85d 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,15 +1,47 @@ -# Beacon primary roadmap: 2.1–2.2 +# Beacon primary roadmap: 2.1–2.5 **Primary planning authority — adopted 4 October 2026.** This is the roadmap Torchlight and the Canadaverse preview work should use going forward. It records the operator-supplied planning discussion and MrAlderson's follow-up. It supersedes the older parity roadmap's release priorities, not its historical evidence. -These are agreed goals, **not a declaration that the features are implemented, -hardware-qualified, accepted upstream or deployed**. Track completion against -reviewable code, tests and the actual environment. Keep the 2.1 scope focused. +The [owner-delegated version allocation](https://discord.com/channels/1507764602253869197/1507788454774182030/1556399330699640893) +was added by **[Torchlight]** at the +[owner's request](https://discord.com/channels/1507764602253869197/1507788454774182030/1556400784885809313). +These are agreed planning targets, **not a declaration that features are implemented, +hardware-qualified, accepted upstream or deployed**, nor fixed delivery dates. +Track completion against reviewable code, tests and the actual environment. Keep +2.1 focused; move a target when its evidence, compatibility or qualification requires it. -## 2.1.0 — Foundations +## Versioning and status rules + +- **🟢 KEEP / EXISTS:** useful or already present, not a blanket stability claim. + **🟡 TARGET / VERIFY / DESIGN:** work or qualification remains. + **🔴 NO SLOT:** a rejected duplicate/default, or work explicitly outside this plan. +- Small compatible fixes and enhancements can use patch releases. Substantial new + workflows or coordinated required API changes belong at a minor-version boundary. + Reserve major-version review for genuinely incompatible changes, not feature size; + an additive database migration alone does not require 3.0 or a fresh-database reset. +- This allocation is informed by Beacon's history, not a claim of a comprehensive + formal SemVer policy: [web 1.3.0](https://github.com/MeshCore-Beacon/beacon-web/releases/tag/v1.3.0) + and [server 1.6.0](https://github.com/MeshCore-Beacon/beacon-server/releases/tag/v1.6.0) + bundled substantial features; 2.0 introduced an incompatible database baseline; + [web 2.0.1](https://github.com/MeshCore-Beacon/beacon-web/commit/7fb24789d1649aa079e4568d335d60b3400d87be) + added Swedish and [server 2.0.1](https://github.com/MeshCore-Beacon/beacon-server/commit/0dca03cd4dae1b6061df5078a4390be33b9ed475) + corrected catalogue refresh/defaults. +- The [upstream release policy](https://github.com/MeshCore-Beacon/beacon-docs/blob/f0d5632ad9833305745f54ff1bb2c0e2bb413f8f/docs/releases.md) + keeps server and web on the same **major.minor**, with independent patch numbers. + A web patch must not unexpectedly require a newer server patch: provide graceful + fallback, or introduce the required contract at a minor boundary. Apply the same + discipline to the incremental collection work. API-path versioning is separate. +- Patch numbers below are movable working slots, not simultaneous server/web tag + promises. Release-blocking or urgent fixes go into the earliest safe release, + including 2.1.0 where needed, rather than waiting for an allocated number. + Official release tags and upstream acceptance remain maintainer-controlled. +- Preview versions such as `2.2.1-n30nex.1` do not mean upstream 2.2 has shipped. + R/D/X identifiers refer to the [feature decision register](docs/post-20-feature-decisions-20261004.md). + +## 🟡 2.1.0 — Foundations - Add the basic **Atlas and Topology pages**, using the existing preview as the starting point rather than introducing a competing workflow. @@ -25,9 +57,21 @@ reviewable code, tests and the actual environment. Keep the 2.1 scope focused. - Preserve the current Topology approach as the baseline; gather specific feedback on appearance and interaction instead of expanding this release into a redesign. -## 2.1.1–2.1.9 — Telemetry and collection in Atlas +## 🟡 2.1.1 — Correctness + +- Target confirmed WebSocket-loss/backoff/write-lifecycle fixes (R01), API-filter + and input validation (R02), TRACE correctness (R03), unclipped/bounded trace rows + and reliable error states (the corrective parts of R04/R07). +- Reproduce individual cases and preserve existing working mechanisms; the supplied + B1–B9 checklist is not nine automatically confirmed bugs. Keep changes compatible. +- Release-blocking fixes belong in 2.1.0 rather than waiting. This correctness slot + does not displace the agreed collection programme or approve unrelated features. + +## 🟡 2.1.2–2.1.9 — Telemetry and collection in Atlas - Expand telemetry collection and its presentation within Atlas incrementally. + These are working slots within the adopted 2.1.x collection programme, not a + requirement to publish every numbered patch or defer work already safe to deliver. - The **mobile application is an opt-in collector**, connecting to a companion radio over BLE and running in the phone background where the operating system permits. Background/BLE reliability must be qualified, not assumed. @@ -44,21 +88,49 @@ reviewable code, tests and the actual environment. Keep the 2.1 scope focused. view, with freshness, units, sensor channel and visible gaps. - Coordinate the mobile client, desktop collector, Atlas and server API as one end-to-end feature, retaining their separate responsibility boundaries. +- Keep collector safety, reconnect reliability and truthful missing-data handling + ahead of unrelated routing/search work (R05/R07/D05). Preserve cross-patch + compatibility using the established intake contract, optional fields and fallbacks. -## 2.2 — Complete Atlas and refine Topology +## 🟡 2.2.0 — Complete Atlas and refine Topology - Complete the agreed Atlas experience, including accepted telemetry and collection integration. - Refine Topology using observed usability feedback and the current implementation. - Fix major and minor defects and address measured performance problems across - the accepted system. Broader feature ideas remain separately prioritized backlog. + the accepted system. +- Target the coordinated additions at this minor boundary: bounded cached topology + links (R06), resolved packet/trace-list summaries (R04), configurable current + IATA-membership freshness that preserves history (R09), and focused API-contract + checks/live-cache consistency (D04). +- Qualify API/UI pairs, query costs, caps, stale/unknown data and compatibility. + Undirected topology links are not directional SNR measurements or a route planner. + +## 🟡 2.2.1 — Small usability additions -## Future interoperability — firmware-direct telemetry (FW01) +- Contact QR/deep links and bundled radio-preset names with raw-setting fallback (R08). +- URL-persisted filters, keyboard/accessibility improvements and mobile sorting over + already-supported server ordering (D04). +- Reuse existing components and pending contributions rather than duplicating them. + A new mandatory backend sorting contract belongs in the later minor release. -**🟢 Keep the direction; 🟡 design and integration remain. No release assigned.** +## 🟡 2.2.2 — Optional visual polish + +- Mini-maps, legends, theme-aware maps, saved display preferences and bounded, + optional Public chatter (D06), using existing APIs and honest position labels. +- Measured rendering improvements and low-power behavior, not an unrequested UI rewrite. +- Do not invent SNR quality; directional SNR-coloured links wait for the evidence + model in 2.3.0. Optional polish is not a blocker for core 2.2.0 acceptance. + +## 🟡 2.2.3 — Firmware interoperability pilot (FW01, conditional) + +**🟢 Keep the direction; 🟡 partner/contract qualification remains.** The [owner's 4 October clarification](https://discord.com/channels/1507764602253869197/1507788454774182030/1556384736522281032) keeps collaboration with repeater and observer firmware projects open, including projects like [MeshCore Observer](https://observer.gessaman.com/). +The earlier unassigned goal now has a **conditional 2.2.3 pilot target**, not a +partner commitment or date. If it needs a new mandatory server contract, move it +to the next minor release; if partners are not ready, defer without blocking 2.2. Wi-Fi-capable devices should be able to opt in and configure a Beacon deployment's telemetry collection endpoint, submitting their own readings through the shared, @@ -84,8 +156,50 @@ Before claiming support, agree and qualify: The reference project's public setup page documents Wi-Fi and MQTT configuration; that is not verification of native Beacon telemetry-API support or a collaboration agreement. No partner outreach or firmware qualification has occurred in this update. -This future direction does not add a new 2.1/2.2 delivery requirement, authorize -firmware changes or imply remote console/control support. +This is not a core 2.1/2.2.0 acceptance requirement, authorization to modify firmware, +or remote console/control support. + +## 🟡 2.2.4+ — Maintenance + +Fix defects, tune measured performance and deliver compatible refinements. Do not +use patch releases as a dumping ground for new database-heavy subsystems or let +these placeholder numbers delay an urgent fix. + +## ❕ 🟡 2.3.0 — Evidence and investigation (new minor) + +- Directional SNR aggregation with sample count/age, direct/inferred provenance, + edge aging, movement invalidation and global versus contextual ambiguity (D01). +- Bounded indexed transit-node history and retained-history search, broader + server-side sorting/keysets, and node/trace investigation capabilities (D03). +- Distinguish origin reports from attributed relay evidence and requested TRACE + paths. Define retention/counting, indexes, migration/recovery and production-like + query costs before selecting implementations. Terminal receiver SNR is not every hop. +- Evidence-backed SNR styling may follow only after the data model is qualified. + Scope each contribution; this target is not permission for a monolithic rewrite. + +## ❕ 🟡 2.4.0 — Calculated routing (new minor) + +Best/alternative routes and MeshCore export (D02) follow the qualified 2.3 evidence +model and bounded graph/query design. Show stale snapshots, retain observed-route +history and test supported client exports. A proposed route is not proof of RF +delivery. This release does not depend on completing replay or unrelated analytics. + +## ❕ 🟡 2.5.0 — Replay and deeper analysis (later review target) + +Historical playback/seek, geographic analysis and deeper fleet/channel analytics +are a later, separately scoped review target. They require retained/indexed evidence, +honest coverage/gaps and measured query/rendering cost. Hourly aggregates cannot +reconstruct expired packet paths. Keep these workflows separable from routing. + +## 🔴 No slot, or explicitly undecided + +- X01 already-covered capabilities and X02 unsuitable imports receive no new port + slot. Keep working upstream modules; reject their duplicates, not the capabilities. +- Remote administration, privileged owner-data feeds and firmware-region catalogue + integration remain separately undecided pending product/privacy/producer contracts. +- No 3.0 is scheduled merely because the backlog is large. A genuinely incompatible + API, data or configuration change requires an explicit major-version decision and + upgrade/recovery design; no fresh-database reset is planned by this document. ## Collector safety and acceptance gates @@ -121,19 +235,23 @@ This document does not itself enable a workflow, merge upstream or deploy to Torchlight may carry out authorized preview-fork development under its existing standing authority. It should use this roadmap to prioritize and maintain task -states, checkpoints, issues/PRs, source revisions and preview verification. Future -ideas do not automatically become 2.1 requirements. Escalate scope changes instead -of silently adding them to a release. +states, checkpoints, issues/PRs, source revisions and preview verification. Planning +allocation does not authorize starting every feature. Escalate scope/compatibility +changes instead of silently adding them to a release. ## Supporting evidence and history - [Current preview](https://canadaverse.org/beacon-dev/) and [published build metadata](https://canadaverse.org/beacon-dev/build.json). -- [Torchlight project dashboard](https://canadaverse.org/torchlight/). +- [Torchlight Dashboard — Beacon Project Tracker](https://canadaverse.org/torchlight/). - [Collector research and earlier design discussion](docs/beacon-21-collector-design.md): useful implementation evidence; conflicting earlier priorities yield to this roadmap. +- [Feature decision register](docs/post-20-feature-decisions-20261004.md): source-backed + R/D/X recommendations and acceptance gaps; the allocation above now supplies targets. - [Broader parity backlog](docs/post-140-roadmap.md): supporting backlog, not the default release sequence. +- [Roadmap before this version allocation](https://github.com/n30nex/beacon-docs-contributions/blob/41b51adbad048f69e4059d06db1c4aae2154de46/ROADMAP.md) + preserves the original 2.1–2.2 plan and initially unscheduled FW01 clarification. - [Previous roadmap and dated checkpoints](https://github.com/n30nex/beacon-docs-contributions/blob/483e71188fb000eeb33db080936573c06e96f8a2/ROADMAP.md) remain preserved in Git history. Old deployment, version and rollback statements are historical and must not be applied as current operating instructions. From ccf67b1eea1685319a4fa2bf7703aee411208fa1 Mon Sep 17 00:00:00 2001 From: MrAlders0n <55921894+MrAlders0n@users.noreply.github.com> Date: Sun, 4 Oct 2026 23:22:36 -0400 Subject: [PATCH 66/69] docs(meshmapper): document API key setup (#14) --- README.md | 5 ++ app_config/.env.example | 7 +++ app_config/config.yaml.example | 10 +++- docker-deployment-type1/data/app/config.yaml | 10 +++- docker-deployment-type2/README.md | 5 ++ .../server/data/app/config.yaml | 10 +++- docs/backup-export.md | 4 ++ docs/configuration.md | 55 ++++++++++++++++++- docs/security.md | 13 ++++- docs/troubleshooting.md | 31 +++++++++++ docs/upgrading.md | 25 ++++++++- 11 files changed, 168 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 7404752..f7d9a4b 100644 --- a/README.md +++ b/README.md @@ -71,6 +71,11 @@ Everything else in the file is optional and explained in [Configuration](docs/configuration.md): the admin API key, log settings, Redis, and the web app's map view, tabs, themes, name and banner. +If you enable MeshMapper imports, also set `MESHMAPPER_API_KEY`. Your local MeshMapper +regional or grouped-region admin can generate a regional key or a key covering multiple +regions. See [MeshMapper API key](docs/configuration.md#meshmapper-api-key) for setup and +the server rollout requirements. Keep this key out of `VITE_*` settings. + ### 3. Describe your network in `config.yaml` ```bash diff --git a/app_config/.env.example b/app_config/.env.example index 41353e2..d85bd62 100644 --- a/app_config/.env.example +++ b/app_config/.env.example @@ -15,6 +15,13 @@ LISTEN_ADDR=:8080 # At least 16 characters, no whitespace; e.g. `openssl rand -hex 32`. # BEACON_API_KEY= +# MeshMapper regional or grouped-region API key for enabled imports. +# Your local MeshMapper regional or grouped-region admin can generate the key; +# grouped-region keys cover multiple regions. Never use the mobile App key. +# Overrides meshmapper.api_key, even when empty. Keep this out of VITE_* settings. +# Use the environment setting with backups; saved YAML keys prevent exports. +# MESHMAPPER_API_KEY= + # Override log.level / log.format from config.yaml (defaults: info / text). # LOG_LEVEL=info # LOG_FORMAT=json diff --git a/app_config/config.yaml.example b/app_config/config.yaml.example index 6491285..c30fe48 100644 --- a/app_config/config.yaml.example +++ b/app_config/config.yaml.example @@ -45,7 +45,14 @@ ratelimit: # A 429 returns error.code=rate_limited and Retry-After (1 or 60 seconds). burst: 300 -# MeshMapper integrations (optional, no API key). Import transport scopes and +# MeshMapper integrations (optional, require a regional or grouped-region API key). +# Your local MeshMapper regional or grouped-region admin can generate the key. +# A grouped-region key covers multiple regions, including imported group members. +# MESHMAPPER_API_KEY overrides meshmapper.api_key, even when empty. Use the +# environment for secrets, especially with backups; never use the mobile App key. +# Missing keys leave imports unconfigured; cached data and the rest of Beacon remain. +# 401/403 failures retain cached data and report authentication/permission errors. +# Import transport scopes and # IATA border outlines from MeshMapper instead of maintaining scopes: and # iatas.*.borderFile by hand. # @@ -63,6 +70,7 @@ ratelimit: # Region-filtered scope lists and scope stats show imported names only under the # IATAs whose catalogue lists them; IATAs without a catalogue show manual scopes only. #meshmapper: +# api_key: "" # scopes: # enabled: false # refresh_interval: 1h # 1h-24h (MeshMapper allows one get_scopes.php per region per 55m); one source per 15s tick diff --git a/docker-deployment-type1/data/app/config.yaml b/docker-deployment-type1/data/app/config.yaml index 6dfc478..59f3592 100644 --- a/docker-deployment-type1/data/app/config.yaml +++ b/docker-deployment-type1/data/app/config.yaml @@ -46,7 +46,14 @@ ratelimit: # A 429 returns error.code=rate_limited and Retry-After (1 or 60 seconds). burst: 300 -# MeshMapper integrations (optional, no API key). Import transport scopes and +# MeshMapper integrations (optional, require a regional or grouped-region API key). +# Your local MeshMapper regional or grouped-region admin can generate the key. +# A grouped-region key covers multiple regions, including imported group members. +# MESHMAPPER_API_KEY overrides meshmapper.api_key, even when empty. Use the +# environment for secrets, especially with backups; never use the mobile App key. +# Missing keys leave imports unconfigured; cached data and the rest of Beacon remain. +# 401/403 failures retain cached data and report authentication/permission errors. +# Import transport scopes and # IATA border outlines from MeshMapper instead of maintaining scopes: and # iatas.*.borderFile by hand. # @@ -64,6 +71,7 @@ ratelimit: # Region-filtered scope lists and scope stats show imported names only under the # IATAs whose catalogue lists them; IATAs without a catalogue show manual scopes only. #meshmapper: +# api_key: "" # scopes: # enabled: false # refresh_interval: 1h # 1h-24h (MeshMapper allows one get_scopes.php per region per 55m); one source per 15s tick diff --git a/docker-deployment-type2/README.md b/docker-deployment-type2/README.md index de6a7ff..ec620a3 100644 --- a/docker-deployment-type2/README.md +++ b/docker-deployment-type2/README.md @@ -45,6 +45,11 @@ docker-deployment-type2/ Caddy redirects anything that is not `/api/*` or `/ws` to that host. The password inside `POSTGRES_DSN` must match `POSTGRES_PASSWORD` in `docker-compose.yml`. + For MeshMapper imports, set `MESHMAPPER_API_KEY` in this server host's `.env`. + Your local MeshMapper regional or grouped-region admin can generate the regional or + grouped-region key. Do not copy it to the web host or a `VITE_*` setting. See + [MeshMapper API key](../docs/configuration.md#meshmapper-api-key). + 3. Edit `data/app/config.yaml` for your network (see [Configuration](../docs/configuration.md#configyaml)). Because the web app will be served from another origin, browsers may only open the WebSocket if that origin is listed: diff --git a/docker-deployment-type2/server/data/app/config.yaml b/docker-deployment-type2/server/data/app/config.yaml index 6dfc478..59f3592 100644 --- a/docker-deployment-type2/server/data/app/config.yaml +++ b/docker-deployment-type2/server/data/app/config.yaml @@ -46,7 +46,14 @@ ratelimit: # A 429 returns error.code=rate_limited and Retry-After (1 or 60 seconds). burst: 300 -# MeshMapper integrations (optional, no API key). Import transport scopes and +# MeshMapper integrations (optional, require a regional or grouped-region API key). +# Your local MeshMapper regional or grouped-region admin can generate the key. +# A grouped-region key covers multiple regions, including imported group members. +# MESHMAPPER_API_KEY overrides meshmapper.api_key, even when empty. Use the +# environment for secrets, especially with backups; never use the mobile App key. +# Missing keys leave imports unconfigured; cached data and the rest of Beacon remain. +# 401/403 failures retain cached data and report authentication/permission errors. +# Import transport scopes and # IATA border outlines from MeshMapper instead of maintaining scopes: and # iatas.*.borderFile by hand. # @@ -64,6 +71,7 @@ ratelimit: # Region-filtered scope lists and scope stats show imported names only under the # IATAs whose catalogue lists them; IATAs without a catalogue show manual scopes only. #meshmapper: +# api_key: "" # scopes: # enabled: false # refresh_interval: 1h # 1h-24h (MeshMapper allows one get_scopes.php per region per 55m); one source per 15s tick diff --git a/docs/backup-export.md b/docs/backup-export.md index 355ef37..0bea075 100644 --- a/docs/backup-export.md +++ b/docs/backup-export.md @@ -22,6 +22,10 @@ changes, PostgreSQL roles and cluster settings, Redis and deployment files are n ## Requirements and limits +- Keep the MeshMapper API key in `MESHMAPPER_API_KEY`, with `meshmapper.api_key` empty or + omitted in the saved YAML. Both CLI exports and admin downloads refuse a nonempty saved + MeshMapper key because the YAML is included verbatim. Environment secrets remain excluded; + preserve them separately for recovery. See [MeshMapper API key](configuration.md#meshmapper-api-key). - `pg_dump` must be in the **same runtime** as the exporter (the CLI's environment, or the server's PATH for the download), with a major version at least as new as the database's. A client on the Docker host or in another container doesn't count. diff --git a/docs/configuration.md b/docs/configuration.md index 9d12ac0..24c0690 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -23,6 +23,7 @@ to open). | `MQTT_BROKER_1_PASSWORD` | unset | Subscriber password for broker 1. | | `MQTT_BROKER_2_URL`, `_USERNAME`, `_PASSWORD` | unset | The same for an optional second broker. A packet heard through both brokers is stored once. | | `BEACON_API_KEY` | unset | Bearer key for `/api/v1/admin/*`. When set, even to an empty string, it replaces `auth.api_key` from the config file. See [Admin API key](#admin-api-key). | +| `MESHMAPPER_API_KEY` | unset | Regional or grouped-region key for enabled MeshMapper imports. When set, even to an empty string, it replaces `meshmapper.api_key`. Missing keys leave imports unconfigured while Beacon keeps running. See [MeshMapper API key](#meshmapper-api-key). | | `LOG_LEVEL` | `info` | `debug`, `info`, `warn` or `error`. Overrides `log.level`. Anything else stops startup. | | `LOG_FORMAT` | `text` | `text` or `json`. Overrides `log.format`. Anything else stops startup. | | `BEACON_CPU_PROFILE_DIR`, `BEACON_CPU_PROFILE_UNTIL` | unset | Enable a bounded CPU capture in production. See [Profiling](profiling.md). | @@ -68,7 +69,7 @@ which ones to read. | `log` | Log level and format. | You want JSON logs for a collector. | | `server` | `trusted_proxies`: the proxy addresses allowed to tell Beacon the real client IP. | Always, if Beacon sits behind a proxy. See [Reverse proxy](reverse-proxy.md). | | `ratelimit` | Per-client REST limits for `/api/v1/*`. On by default at 300 requests a minute. | You get 429s you did not expect. | -| `meshmapper` | Import transport scopes, IATA borders, region groups and public channels from MeshMapper instead of listing them by hand. | You are in a region MeshMapper covers. | +| `meshmapper` | Import transport scopes, IATA borders, region groups and public channels from MeshMapper instead of listing them by hand. Requires a [MeshMapper API key](#meshmapper-api-key). | You are in a region MeshMapper covers. | | `iatas` | Display names, coordinates and optional border files for airport codes. IATAs are created automatically when traffic arrives; this block only decorates them. A `borderFile` path is relative to `config.yaml`; in Docker the compose file mounts only `config.yaml`, so add a mount for the folder too (`- ./data/app/borders:/app/borders:ro` under the `app` service). | You want names on the map or a border drawn. | | `regions` | Groups of IATAs with a name, map centre and zoom. | Always, unless MeshMapper imports them. | | `channel_keys` | Hashtag channels and explicit keys for decrypting group messages. | Always. Without keys, channel messages are stored as hashes only. | @@ -106,6 +107,58 @@ cannot honour. These are the ones people hit: A negative `websocket.max_connections_per_ip` is not caught at startup and rejects every connection instead, so leave it at a positive number. +## MeshMapper API key + +MeshMapper APIs require either a regional API key or a grouped-region API key covering +multiple regions. Your local MeshMapper regional or grouped-region admin can generate +these keys. Request a key for the APIs Beacon uses, covering your IATA or every member IATA +in your group, including members added by `meshmapper.zones.import_groups`. + +Set `MESHMAPPER_API_KEY` in the deployment's private `.env` or backend environment. Each +Beacon deployment can use a different key. It overrides `meshmapper.api_key` in +`config.yaml`, even when the environment value is empty. Both examples leave the key empty. +Do not use the mobile app's App key, `BEACON_API_KEY`, or a Coverage API key for this setting. +Never put it in a `VITE_*` value, browser configuration, logs, or git. + +After changing `.env`, recreate the backend container from the deployment folder +(`docker-deployment-type2/server` for the split deployment): + +```bash +docker compose up -d --force-recreate app +``` + +A plain `docker compose restart app` does not load changed container environment variables. +If you use the YAML setting instead, restarting Beacon is enough. Prefer the environment +setting when using [backups](backup-export.md): exports refuse a nonempty +`meshmapper.api_key` in the saved YAML so the key cannot enter a download. + +Beacon sends `X-API-Key` to `get_zones.php`, `get_geojson.php`, `get_scopes.php`, and +`get_channels.php`. Its shared client supports the same header for `get_repeaters.php`, +although Beacon does not currently fetch repeaters. Requests use trusted HTTPS MeshMapper +endpoints and do not follow redirects. `get_zones.php` still needs its `country` parameter; +a key covering multiple regions does not change the request shape or remove rate limits. +IP exemptions are not authentication. + +With no key, MeshMapper imports report unconfigured and make no requests. Beacon keeps +running and restores saved imports. Failed refreshes keep the last successful cached data +and its freshness timestamp; 401 reports authentication failure and 403 reports permission +failure, without falling back to anonymous requests. Existing backoff and longer +`Retry-After` delays on 429/503 remain in effect. See +[MeshMapper troubleshooting](troubleshooting.md#meshmapper-imports-are-unconfigured-or-stale). + +### Coordinate the MeshMapper rollout + +Before enforcement, confirm with the MeshMapper admin that Server supports `X-API-Key`, +that the key covers the required APIs and IATAs, and that these calls do not consume Coverage +quota. Header support must not be assumed deployed. The first four endpoints can require +keys before the app rollout; repeater enforcement waits for app 1.4.1 and its forced update. +Key issuance, permissions, quotas, and enforcement switches remain MeshMapper Server work, +tracked in [MeshMapper_Server#431](https://github.com/MeshMapper/MeshMapper_Server/issues/431). + +Existing `coverage.php` consumers keep their own keys, scopes, quotas, and supported +`?key=` authentication. Do not move them to header-only authentication without confirmed +server support; Beacon's importer does not call `coverage.php`. + ## Admin API key The `/api/v1/admin/*` endpoints need `Authorization: Bearer `. Everything else, including diff --git a/docs/security.md b/docs/security.md index 05c68fb..85b6bfc 100644 --- a/docs/security.md +++ b/docs/security.md @@ -1,7 +1,8 @@ # Securing a deployment Beacon is a read-mostly public service. The things worth protecting are the admin key, the API -listener, your broker credentials, and the database, which holds decrypted channel messages. +listener, your broker and MeshMapper credentials, and the database, which holds decrypted +channel messages. ## The admin key @@ -14,6 +15,14 @@ environment wins over the file) are in [Configuration](configuration.md#admin-ap If you do not need the admin endpoints, do not set a key. They answer `503` and nothing else changes. +## The MeshMapper API key + +Obtain a regional or grouped-region API key from your local MeshMapper admin and keep it +in `MESHMAPPER_API_KEY` in the backend environment. Never use the mobile App key, publish +the key through a `VITE_*` setting, or commit it. Beacon sends it only as `X-API-Key` to +the supported HTTPS MeshMapper API endpoints and refuses redirects. This is separate +from Beacon's admin authentication. See [MeshMapper API key](configuration.md#meshmapper-api-key). + ## Keep the API listener private beacon-server listens on `:8080` with no TLS. In the Docker deployments it is only reachable @@ -49,6 +58,6 @@ if you enable it, contains everything the database holds, so keep archives priva ## Secrets in files -`.env` holds the database password, broker credentials and the admin key. It is gitignored in +`.env` holds the database password, broker credentials, admin key, and MeshMapper API key. It is gitignored in the deployment folders; keep it that way, and keep its permissions tight on the host. `data/app/config.yaml` holds channel keys. Neither belongs in a public backup. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 4d29677..f4840a5 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -93,6 +93,37 @@ The channel's key is not configured, or it was added without a restart. Add it u the stored messages for the new key and logs `config: backfilled N previously-undecrypted channel message(s)`. +## MeshMapper imports are unconfigured or stale + +Read `docker compose logs app` for `component=meshmapper`, `meshmapper.zones`, +`meshmapper.scopes`, or `meshmapper.channels`. Refresh records include `last_error`, +`next_attempt`, and, for regional imports, `checked_at`. + +| Message | Fix | +|---|---| +| `unconfigured: MeshMapper API key is missing` | Set `MESHMAPPER_API_KEY` in the backend environment. An explicitly empty environment value overrides a YAML key. | +| `unconfigured: MeshMapper API key contains invalid characters` | Re-enter the key supplied by your local MeshMapper regional or grouped-region admin; remove embedded whitespace or control characters. | +| `authentication failed (HTTP 401)` | Check that you are using the correct regional or grouped-region API key and that MeshMapper Server supports `X-API-Key`. Do not use the mobile App key. | +| `permission denied (HTTP 403)` | Ask the MeshMapper admin to check access to the requested API and IATA. A group key needs every member IATA, including members added by `import_groups`. | +| `HTTP 429` or `HTTP 503` | Wait until `next_attempt`. Beacon honours longer `Retry-After` delays and its existing refresh limits. Do not shorten polling intervals or repeatedly restart to force a request. | + +After changing `.env`, run `docker compose up -d --force-recreate app` from the backend +deployment folder. A plain restart does not reload container environment variables. +Persisted backoff still applies after a restart. Never paste the key into logs, URLs, or +support messages. + +Beacon keeps the last successful cached data and freshness timestamp after a failed +refresh, including an authentication failure. It does not replace imports with empty +lists or retry anonymously. See [MeshMapper API key](configuration.md#meshmapper-api-key) +for setup and rollout requirements. + +## Backup export refuses a saved MeshMapper key + +Move the key from `meshmapper.api_key` to `MESHMAPPER_API_KEY` in the backend environment, +clear the saved YAML value, and recreate `app`. Backups include saved YAML verbatim, so +exports refuse a nonempty MeshMapper key there to keep it out of download responses. +See [Backup and export](backup-export.md#requirements-and-limits). + ## I changed a `VITE_*` value and nothing changed Run `docker compose up -d web` (`docker compose up -d beacon-web` in the split deployment). The web container writes the values to `/config.js` when it diff --git a/docs/upgrading.md b/docs/upgrading.md index 7b46061..3f4c5f9 100644 --- a/docs/upgrading.md +++ b/docs/upgrading.md @@ -93,6 +93,26 @@ docker compose up -d Database migrations run when the server starts. Read the release notes first; a release that needs a config change says so there. Keep server and web on the same `X.Y`. +## MeshMapper API authentication rollout + +When upgrading to a Beacon build with MeshMapper API authentication, configure the key +before relying on imports. Ask your local MeshMapper regional or grouped-region admin +for a key covering the required APIs and your region or all member regions. Confirm +MeshMapper Server supports `X-API-Key` before enforcement; see +[MeshMapper API key](configuration.md#meshmapper-api-key) for the rollout details. + +Set `MESHMAPPER_API_KEY` in the backend `.env`, then recreate the `app` container: + +```bash +docker compose up -d --force-recreate app +``` + +Without a key, Beacon keeps running and retains saved imports, but MeshMapper refreshes +are unconfigured. Existing IP exemptions are not a substitute for a key. After configuring +one, check the MeshMapper logs for successful refreshes; existing backoff still applies. +If you use backups, leave `meshmapper.api_key` empty in the saved YAML and keep the key in +the environment. Exports refuse a nonempty saved YAML key. + ## Changes that need a restart Beacon reads `config.yaml` once at startup. After editing it, run @@ -101,7 +121,10 @@ Beacon reads `config.yaml` once at startup. After editing it, run - A newly added channel key decrypts the matching stored messages. Look for `config: backfilled N previously-undecrypted channel message(s)` in the log. - Border file and MeshMapper import changes take effect. -- A changed admin key is picked up. +- Changed admin or MeshMapper keys in the YAML file are picked up. + +For environment changes in `.env`, use `docker compose up -d --force-recreate app` instead. +A container restart reuses its old environment. The one runtime-only setting is CORS origins: `PUT /api/v1/admin/config` changes them without a restart, and nothing is written to the file, so a restart reloads whatever the file says. From 791417368b11ba9e4e3e6d7899b56bab45bf90a9 Mon Sep 17 00:00:00 2001 From: MrAlders0n <55921894+MrAlders0n@users.noreply.github.com> Date: Mon, 5 Oct 2026 06:34:00 -0400 Subject: [PATCH 67/69] docs(observers): document directory pagination (#15) --- docs/api-contract.md | 1 + docs/observer-directory.md | 118 +++++++++++++++++++++++++++++++++++++ 2 files changed, 119 insertions(+) create mode 100644 docs/observer-directory.md diff --git a/docs/api-contract.md b/docs/api-contract.md index 32a0b0c..48243f3 100644 --- a/docs/api-contract.md +++ b/docs/api-contract.md @@ -277,6 +277,7 @@ edges spanning more than 180 degrees are rejected. | Endpoint | Notes | |---|---| | `GET /observers` | Page of observers. Params: location filters, `type` (e.g. `meshcoretomqtt`), `broker`, `status` (`online`/`offline`), `name`, `scope`, `cursor` (`lastSeen` epoch ms), `limit`. | +| `GET /observers/directory` | Traffic/name pagination, row counts and type facets. See [observer directory contract](observer-directory.md) for rollout status and exact semantics. | | `GET /observers/{observerId}` | Observer detail. | | `GET /observers/{observerId}/telemetry?range=24h&interval=1h` | Telemetry history. `interval` is `1h` (default), `6h` or `24h`. `afterId` returns only newer points. | | `GET /observers/{observerId}/activity?range=24h&interval=15m` | What the observer heard, bucketed. | diff --git a/docs/observer-directory.md b/docs/observer-directory.md new file mode 100644 index 0000000..550df25 --- /dev/null +++ b/docs/observer-directory.md @@ -0,0 +1,118 @@ +# Observer directory + +Status: pending [server PR #212](https://github.com/MeshCore-Beacon/beacon-server/pull/212), +not yet deployed. The existing `/observers` endpoint remains unchanged. + +## Requests + +`GET /api/v1/observers/directory?iata=YVR&sort=traffic&limit=50` + +| Parameter | Contract | +|---|---| +| `sort` | `traffic` (default) or `name`. | +| `limit` | Positive integer, default 50, capped at 200 per page. No directory-wide row cap. | +| `cursor` | Nonnegative integer offset, default 0. Pass the returned `nextCursor`. | +| `since`, `until` | Positive epoch milliseconds, inclusive start and exclusive end, rounded down to UTC hours. Window must be 1 hour through 31 days; `until` cannot be in the future. Required when cursor is nonzero. | +| `iata`, `iatas` | Case-insensitive IATA or comma-separated IATAs. Nonempty `iatas` takes precedence. | +| `regionId`, `region` | Expand region ID or slug and union with explicit IATAs. ID takes precedence. An empty region without explicit IATAs matches nothing. | +| `name` | Case-insensitive display-name substring using SQL ILIKE semantics (`%` and `_` are wildcards). | +| `type`, `broker`, `scope` | Exact observer type, broker membership or transport scope name. URL-encode `#` in scopes. | +| `status` | `online` or `offline`. | + +All filters apply before ordering and pagination. Type, broker, name and scope +are limited to 200 UTF-8 bytes each. At most 200 IATAs are accepted after region +expansion and before deduplication. Unknown/repeated parameters return 400. + +The first page defaults to `until = floorToUtcHour(now - 95 minutes) + 1 hour`, +and `since = until - 24 hours`. This excludes the latest 35 to 95 minutes; +these are hourly analytics counts, not live packet counters. + +For every continuation, repeat all filters and sort, pass the first response's +`windowStart` as `since` and `windowEnd` as `until`, and use `nextCursor`: + +```text +/api/v1/observers/directory?iata=YVR&sort=traffic&since=1767225600000&until=1767312000000&cursor=50&limit=50 +``` + +## Response + +Items contain the existing `ObserverSummary` fields plus required nullable +`observationCount`. Optional names, type, radio and empty scopes may be omitted. + +```json +{ + "items": [ + {"id": "00000000-0000-0000-0000-000000000001", "displayName": "North receiver", "iata": "YVR", "status": "online", "observationCount": 1200} + ], + "nextCursor": 1, + "hasMore": true, + "generatedAt": 1767314100000, + "windowStart": 1767225600000, + "windowEnd": 1767312000000, + "sort": "traffic", + "effectiveSort": "traffic", + "coverage": {"status": "complete", "expectedHours": 24, "completeHours": 24, "partialHours": 0, "missingHours": 0}, + "maxObservationCount": 1200, + "observerTypes": ["meshcore-ha", "meshcoretomqtt"] +} +``` + +Timestamps are epoch milliseconds. `items` and `observerTypes` are always arrays. +A terminal page has `hasMore: false` and no `nextCursor`. A cursor past the end +returns an empty terminal page. + +Counts sum `analytics_hourly_observer_identity` for the requested window. They +represent stored hearings per observer, deduplicated across brokers, not MQTT +messages or the observer detail's cumulative presence counter. No raw observation +scan is needed. Directory membership uses current observer IATA; counts include +only observations in selected IATAs, or all IATAs without a location filter. +Broker and scope filter membership, not the source of counted observations. + +When every requested hour is complete, missing observer analytics mean **0**. +If any hour is partial or missing, every count and `maxObservationCount` is +**null**, and `effectiveSort` is `name`. Coverage is `complete` when all hours are +complete, `partial` when some are complete or partial but not all complete, and +`unavailable` when all are missing. Absent rollup records count as missing. +Never render null as zero. + +Traffic order is count descending, then lowercase display name ascending using +PostgreSQL `C` collation, then UUID ascending. Name order uses the last two keys. +Missing names sort as empty strings. Online status means last status or last +seen within five minutes at request time. + +`maxObservationCount` covers the whole filtered result, including in name order. +It is 0 for an empty complete result. `observerTypes` includes all distinct +nonempty types matching every filter except the type filter itself, independent +of pagination. + +## Consistency and client behavior + +Each request reads current data. There is no stored snapshot, expiry, new table +or snapshot capacity limit. Keeping the window fixed avoids hourly window drift, +but analytics corrections and metadata changes can still move rows between +pages, causing repeats or skips. Counts, coverage, type choices and maximum can +also change. This endpoint is a browsing list, not a consistent export. + +Use one infinite query keyed by server, filters, window and sort. A query change +starts from page one and discards stale responses. Preserve server order, allow +one continuation in flight, deduplicate by observer UUID, and stop on +`hasMore=false` or a missing/repeated cursor. Fetch again near the scroll end, +including when the viewport is not filled. No per-row count requests or separate +top-observers request is needed. + +For stable bars while scrolling, retain the first page's maximum and clamp +count/max to [0,1]; reset on refresh. Render unavailable when count or maximum is +null, and zero width when the maximum is zero. If `effectiveSort` changes between +pages as coverage changes, restart from page one rather than mixing orderings. +Live status badges can update separately. Refresh the list to update membership +or order; selection and deep links continue to use UUIDs. + +Invalid parameters and unknown regions return 400. Database failures return 500; +retain loaded rows and offer retry. Successful pages and store errors send +`Cache-Control: no-store`; database reads have a five-second operation budget. + +Probe `/api/v1/observers/directory?limit=1` for compatibility. A supported server +returns the response above. Older servers may return 404, or the legacy detail +route's 400 with message `failed to parse observer UUID`. Other errors do not +prove lack of support. Use an explicitly limited legacy fallback or request a +server upgrade; do not assume the old `/observers` route honors new parameters. From ee088b5dfa391c0e0ecd6c2c650d327933327c43 Mon Sep 17 00:00:00 2001 From: MrAlders0n <55921894+MrAlders0n@users.noreply.github.com> Date: Tue, 6 Oct 2026 15:19:37 -0400 Subject: [PATCH 68/69] docs(mobile): add BEACON Mobile privacy policy and support page (#16) --- docs/mobile/privacy.md | 64 ++++++++++++++++++++++++++++++++++++++++++ docs/mobile/support.md | 33 ++++++++++++++++++++++ 2 files changed, 97 insertions(+) create mode 100644 docs/mobile/privacy.md create mode 100644 docs/mobile/support.md diff --git a/docs/mobile/privacy.md b/docs/mobile/privacy.md new file mode 100644 index 0000000..c64500b --- /dev/null +++ b/docs/mobile/privacy.md @@ -0,0 +1,64 @@ +# BEACON Mobile Privacy Policy + +_Last updated: October 6, 2026_ + +BEACON Mobile ("the app") is an open-source iOS and Android client for Beacon, a +real-time analyzer for MeshCore radio mesh networks. This policy explains what +the app does and does not do with your information. + +## The short version + +- No account, no sign-in. +- No analytics, advertising, tracking or crash-reporting SDKs. +- The app does not access your location, contacts, photos, microphone or + camera. +- Your settings stay on your device. + +## What stays on your device + +The app stores its settings locally, using the platform's standard app storage: +the Beacon servers you have added, your selected region, theme, language and map +preferences. This data never leaves your device unless you back up your device +through Apple or Google. Deleting the app deletes it. + +## Network connections the app makes + +To work, the app connects to: + +1. **The Beacon server you choose.** By default this is `dev.meshcore.ca`; you + can add or switch to any other Beacon server, including one you host. The app + requests packet, node, observer and statistics data from it over HTTPS or + WebSocket. Like any web server, that server sees your IP address and the + requests made. A server you add yourself is run by its own operator, under + that operator's own policies. +2. **Map tile providers.** Map imagery is loaded from + [OpenFreeMap](https://openfreemap.org) and elevation data from the public + AWS Terrain Tiles dataset. These providers see your IP address and the map + areas requested, as with any map. + +The app sends no personal information to these services beyond what any network +request carries, such as your IP address. + +## Mesh data shown in the app + +The app displays radio traffic that MeshCore devices broadcast publicly, as +collected by Beacon servers: node names, public keys, advertised positions, +packet routes, signal readings and messages on public channels. This is not +data about you as an app user. If you operate a MeshCore node and want data +about it removed from a Beacon server, contact that server's operator. For the +default server, open an issue at +[github.com/MeshCore-Beacon/beacon-docs/issues](https://github.com/MeshCore-Beacon/beacon-docs/issues). + +## Children + +The app is not directed at children and collects no personal information from +anyone. + +## Changes + +Updates to this policy will be published on this page with a new date. + +## Contact + +Questions about this policy: [MrAlders0n@smal.ca](mailto:MrAlders0n@smal.ca) +or [open an issue](https://github.com/MeshCore-Beacon/beacon-flutter-app/issues). diff --git a/docs/mobile/support.md b/docs/mobile/support.md new file mode 100644 index 0000000..49dc1ca --- /dev/null +++ b/docs/mobile/support.md @@ -0,0 +1,33 @@ +# BEACON Mobile Support + +BEACON Mobile is the iOS and Android client for Beacon, the real-time MeshCore +packet analyzer. + +## Getting started + +The app connects to the public Beacon server at `dev.meshcore.ca` out of the +box, so live packets, the map and analytics appear right away. To use a +different Beacon server, open **Settings → Servers** and add its address. +Running your own server is covered in the +[deployment guide](https://github.com/MeshCore-Beacon/beacon-docs#readme). + +## Beta testing + +- **iPhone and iPad:** join through the public TestFlight link and send feedback + from the TestFlight app (take a screenshot in BEACON to attach it). +- **Android:** join the Google Play testing program from the opt-in link, then + leave feedback on the app's Play page. + +## Report a problem or request a feature + +[Open an issue on GitHub](https://github.com/MeshCore-Beacon/beacon-flutter-app/issues) +with your device model, OS version, the app version (Settings → About) and what +happened. + +## Contact + +Email [MrAlders0n@smal.ca](mailto:MrAlders0n@smal.ca). + +## Privacy + +See the [privacy policy](privacy.md). From 9d1dae3bf9892d58f8c0762c7c74724ec1e80299 Mon Sep 17 00:00:00 2001 From: MrAlders0n <55921894+MrAlders0n@users.noreply.github.com> Date: Tue, 6 Oct 2026 21:47:26 -0400 Subject: [PATCH 69/69] docs: document GET /info and mobile.min_app_version (#18) * docs: document GET /info and mobile.min_app_version * docs: match /info serverVersion and admin field to the server --- app_config/config.yaml.example | 8 ++++++++ docs/api-contract.md | 28 +++++++++++++++++++++++++++- docs/configuration.md | 2 ++ 3 files changed, 37 insertions(+), 1 deletion(-) diff --git a/app_config/config.yaml.example b/app_config/config.yaml.example index c30fe48..08bee2c 100644 --- a/app_config/config.yaml.example +++ b/app_config/config.yaml.example @@ -271,6 +271,14 @@ websocket: #observers: # delete_after: 720h # opt in to 30 days unseen +# BEACON Mobile settings, published unauthenticated at GET /api/v1/info. +# min_app_version: oldest app release allowed to use this server. Older apps +# show an "update required" screen for this server. Strict X.Y.Z (digits only, +# no "v" prefix or -beta suffix); any other value prevents startup. +# Omitted or empty (default) means no requirement. Takes effect on restart. +#mobile: +# min_app_version: "0.1.1" # default: "" (no requirement) + # CORS configuration. # Controls which origins, methods and headers are allowed for cross-origin requests. # Defaults to allowing all origins with GET/HEAD/OPTIONS if omitted, appropriate diff --git a/docs/api-contract.md b/docs/api-contract.md index 48243f3..4fd8498 100644 --- a/docs/api-contract.md +++ b/docs/api-contract.md @@ -591,13 +591,31 @@ A list item: lastHeardAt, rawPath, resolvedRoute }`. `rawPath` is `[{ hash, snr }]` as received; `resolvedRoute` uses the resolved hop shape from packet detail. +### Server info + +``` +GET /api/v1/info +``` + +Public, no key. Counts against the normal rate limit and is sent with `Cache-Control: no-cache`. + +```json +{ "minAppVersion": "0.1.1", "serverVersion": "2.0.3" } +``` + +`minAppVersion` is the server's `mobile.min_app_version`: the oldest BEACON Mobile release +allowed to use this server, as a strict `X.Y.Z` string, or `null` when the operator has not set +one. `serverVersion` is the server's API version (`X.Y.Z`, no `v`), the same value Swagger shows. Servers +older than this endpoint return `404`. See +[Mobile-specific concerns](#mobile-specific-concerns) for how the app uses it. + ### Admin All under `/api/v1/admin/`, bearer key required (see [Auth](#auth)). | Endpoint | Notes | |---|---| -| `GET /admin/config` | Running CORS settings with defaults applied, `auth.configured`, and `ingest.broker_count` (configured broker workers, not connection status). No credentials, broker addresses, channel material or database settings. | +| `GET /admin/config` | Running CORS settings with defaults applied, `auth.configured`, and `ingest.broker_count` (configured broker workers, not connection status), and `mobile.min_app_version` (read-only; `""` when unset). No credentials, broker addresses, channel material or database settings. | | `PUT /admin/config` | Replaces `cors.allowed_origins` until the next restart. JSON body up to 16 KiB. | | `GET /admin/accounts`, `POST /admin/accounts` | List or create operator account records (name only; not logins). | | `GET /admin/accounts/{id}`, `DELETE /admin/accounts/{id}` | Fetch or deactivate one. | @@ -856,6 +874,14 @@ Flutter on iOS background suspension and Android battery saver will kill the Web The protocol doesn't need to know about backgrounding; the client just treats reconnection as the recovery mechanism. +**Minimum app version.** The app calls `GET /info` on launch, on return to foreground and when +the user switches servers. When the app's own version is lower than `minAppVersion` (compared +numerically, part by part), it blocks that server behind an "update required" screen until the +app is updated. Other servers stay usable. `minAppVersion: null` means no requirement. A `404` +comes from a server that predates the endpoint and also means no requirement. Network errors, +timeouts and other failures do not block: the app keeps using the server and checks again next +time. + --- ## Open questions diff --git a/docs/configuration.md b/docs/configuration.md index 24c0690..c679fe6 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -83,6 +83,7 @@ which ones to read. | `websocket` | Connections per client, upgrade attempts per minute, and which other sites may open `/ws`. | The web app is served from a different host than the API. | | `nodes` | When a node is marked stale, when it is deleted, the clock-drift threshold, and the optional foreign repeater flag. | You want `possiblyForeign` on repeaters. | | `observers` | Optional deletion of observers not seen for a long time. Off by default. | Rarely. | +| `mobile` | `min_app_version`: the oldest BEACON Mobile release allowed to use this server. Older apps show an update screen. Unset by default. | You need users on a newer app release. | | `cors` | Browser cross-origin rules for REST. Default allows any origin, read-only methods. | You are building an admin UI on another origin. | | `cache` | Redis TTLs per response category. | Rarely. | | `ingest` | Only store packets from observers in listed countries or continents. | You run a regional instance and want to ignore the rest of the world. | @@ -102,6 +103,7 @@ cannot honour. These are the ones people hit: - A `borderFile` that is missing or not a valid GeoJSON Polygon or MultiPolygon Feature, or `nodes.mark_foreign: true` with no border source at all. - `log.level` or `log.format` set to anything other than the listed values. +- `mobile.min_app_version` that is not a plain `X.Y.Z` release (`v1.2.3` and `1.2.3-beta` are rejected). - An admin key under 16 characters or containing whitespace. - A negative `ratelimit.requests_per_minute`, `ratelimit.burst` or `websocket.max_connects_per_minute`. A negative `websocket.max_connections_per_ip` is not caught at startup and rejects every