Skip to content

refactor(server): remove the dead POST /api/packets endpoint (#223) - #231

Merged
dborup merged 3 commits into
masterfrom
codex/issue-223-remove-post-packets
Oct 5, 2026
Merged

dborup merged 3 commits into
masterfrom
codex/issue-223-remove-post-packets

Conversation

@dborup

@dborup dborup commented Oct 4, 2026 •

Copy link
Copy Markdown
Owner

Relates to #223

Summary

POST /api/packets ran three INSERTs (transmissions, observers, observations) on the server's mode=ro handle (Kpa-clawbot#1283). In production every call failed with a 500 that carried the raw SQLite error. Nothing in the frontend calls it.

This PR removes the endpoint (option b in the issue) and everything that only existed for it. Ingest stays MQTT → cmd/ingestor.

The following are unchanged: GET /api/packets, POST /api/packets/observations, GET /api/packets/timestamps, GET /api/packets/{id}, GET /api/packets/{hash}/path, POST /api/decode and requireAPIKey.

Removed

  • cmd/server/routes.go: the POST /api/packets route and handlePostPacket, plus the now-unused packetpath import.
  • cmd/server/types.go: PacketIngestResponse.
  • cmd/server/decoder.go: PayloadJSON, whose only caller was the handler. The ingestor's copy stays.
  • cmd/server/openapi.go: the POST /api/packets description.
  • cmd/server/routes_test.go: TestPostPacketPersistsV3Schema. It only passed against a writable test DB.
  • proto/packet.proto: PacketIngestRequest / PacketIngestResponse. The proto/decoded.proto comment no longer lists the endpoint.
  • tools/generate-packets.js: a dev script whose only job was to POST synthetic packets to this endpoint. Nothing references it.
  • Docs:
    • docs/api-spec.md: the section and its TOC link are removed.
    • docs/user-guide/faq.md: Q8 now says that ingest is MQTT + ingestor only and that the server is read-only.
    • BUILD_PLAN.md: the "manual injection" lines and the tool are removed from the tree listing.
  • knownServerWriteSQL: the routes.go: 3 entry is removed, so routes.go is now held to 0 write-SQL literals. Since fix(store): move content-hash migration writes to the ingestor and merge duplicates in memory (#215) #222 merged, hash_migrate.go is also at 0. The map is now backup.go: 1, openapi.go: 1 and ping_score_history.go: 15.

apikey_security_test.go used /api/packets only as a placeholder URL around a stub handler. It now uses /api/perf/reset; the assertions are unchanged.

CHANGELOG.md is not touched, because the fork's master has not updated [Unreleased] for any merged PR.

What POST /api/packets returns now

  • API router (RegisterRoutes): 405. The path still matches the GET route, so gorilla/mux rejects the method before any handler or DB access.
  • Production router (main.go): 200 text/html (index.html). main.go mounts a catch-all PathPrefix("/") SPA handler after the API routes, and gorilla/mux lets a later full match win over a method mismatch. This is the existing fallback for every unmatched /api/* request and is not new in this PR. It is tracked in fix(server): unknown /api/* paths and wrong methods return 200 index.html instead of a JSON 404/405 #233. A test pins the current behaviour, and nothing is written.

Guard

There is one guard: TestServerHasNoPacketTableWrites in readonly_invariant_test.go, which arrived with #222. This PR extends it.

  • Statements: INSERT, INSERT OR …, UPDATE, DELETE and REPLACE.
  • Tables: transmissions, observations, observers, dropped_packets, ping_triggers and route_mask_changes.
  • Scope: every non-test cmd/server file, with no exceptions. The two documented write exceptions do not touch these tables: ping_score_history.go writes its own database, and backup.go only runs VACUUM INTO.

Before this PR, INSERT was only checked in hash_migrate*.go, and the guard's comment named handlePostPacket as a known gap. Every violation is now reported with file, line and statement.

TestServerHasNoPacketTableWritesIsSensitive covers the old migration statements and the three removed INSERTs, and adds negative cases (ping_score_history_entries, observations_archive, observer_neighbors, plain SELECT).

Tests

New in cmd/server/post_packets_removed_223_test.go. Each test uses a seeded file DB opened with OpenDB (mode=ro):

  • TestPostPacketsRemovedReturns405OnReadOnlyDB: a valid key and a decodable body get 405. The body contains no SQLite text, and row counts are unchanged.
  • TestPostPacketsRemovedFallsThroughToSPAInProductionRouter: with the main.go catch-all, the request gets 200 index.html and nothing is written.
  • TestPacketsRoutesSurviveRemoval: the other /api/packets* routes and /api/decode still match.
  • TestOpenAPISpecHasNoPostPackets: /api/spec keeps get and has no post for /api/packets.

New in cmd/server/openapi_test.go:

  • TestOpenAPIDescriptionsHaveRoutes: every key in routeDescriptions() must be a registered method + path. The spec is built by walking the router, so before this test an orphaned description failed nothing.

Extended: TestServerHasNoPacketTableWrites and TestServerHasNoPacketTableWritesIsSensitive, as described under Guard.

Perf

No hot path is touched; this PR only removes code and changes tests.

🤖 Generated with Claude Code

handlePostPacket INSERTed into transmissions, observers and observations
on the server's mode=ro handle (Kpa-clawbot#1283), so in production every call
returned 500 with the raw SQLite error. Nothing in the frontend used it;
ingest is MQTT -> cmd/ingestor.

Removed: the route and handler, PacketIngestResponse, the OpenAPI
description, the proto request/response messages, the api-spec section,
the FAQ claim, the BUILD_PLAN lines, tools/generate-packets.js (its only
job was to POST here), the writable-DB round-trip test, and routes.go
from knownServerWriteSQL.

POST /api/packets now gets 405 from the API router; with main.go's
catch-all SPA handler it falls through to index.html like any other
unmatched path. Both are pinned against an OpenDB (mode=ro) store, with
row counts checked. New guards: no INSERT/REPLACE into the packet tables
anywhere in cmd/server, and every OpenAPI description must name a
registered route.

Relates to #223

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018mVL1VB4jtY7caZQnMWJpW
@adminopenclaw8-sketch

Copy link
Copy Markdown
Collaborator

Review — CS-Macmini PR#231 remove-post-packets — head dbc0426

Dom: APPROVE med nits. The removal is correct and complete in behaviour. The branch must be synced with master before merge (F1); F2 and F3 are small and fit in that sync commit.

Evidence tags: [T] = tested/run by me, [A] = read/analysed in source, [K] = taken from the author's work log or PR text, not verified by me.

There is no author report on this PR. This review is based on issue #223, finding N9 from my #222 review, the PR description and the diff. Items from the author's work log are tagged [K].

Setup: git fetch --no-prune origin; head dbc04260 (unchanged before and after, via git ls-remote); base at branch cut 7697a826; current master aff158c7. I used git archive of the head and of git merge-tree --write-tree origin/master dbc04260… in scratch. No checkout was modified.

Findings

# Sev Finding Evidence
F1 Medium (merge blocker, trivial) The PR conflicts with current master. #222 merged after this branch was cut and rewrote the same comment block in cmd/server/readonly_sql_literal_test.go. GitHub reports mergeable: CONFLICTING. Only the comment conflicts. The knownServerWriteSQL map auto-merges to the right result: neither hash_migrate.go nor routes.go, leaving backup.go: 1, openapi.go: 1, ping_score_history.go: 15. To resolve, keep that map and merge the two comment lines, e.g. "hash_migrate.go had 3 until the content-hash migration moved to the ingestor (#215); routes.go had 3 until POST /api/packets was removed (#223)." All test results below come from a merged tree with exactly that resolution. [T] git merge-tree, gh pr view --json mergeable
F2 Low After the merge, master's guard comment is false, and there are two packet-table guards with different table lists. readonly_invariant_test.go:225-228 (TestServerHasNoPacketTableWrites, from #222) says: "Known gap … handlePostPacket (routes.go) INSERTs into transmissions/observations … INSERT is therefore only checked in the migration files." With this PR that gap is gone. Master's guard covers UPDATE/DELETE/REPLACE on transmissions|observations|ping_triggers|route_mask_changes. The new TestServerHasNoPacketTableInserts covers INSERT/REPLACE on transmissions|observations|observers|dropped_packets. Minimum fix in the sync commit: update that comment to point at the new guard. Better, now or as a follow-up: fold INSERT into the everywhere list of the existing guard and use one union table list (DRY: one guard, one list). The PR text "…from #222, which … is not on master yet" is also stale. [A] read both guards on the merged tree
F3 Low There is a leftover helper. PayloadJSON (cmd/server/decoder.go:724, "serializes the payload to JSON for DB storage") had one caller in cmd/server, handlePostPacket. It now has zero callers and no tests in cmd/server. The ingestor keeps its own copy (cmd/ingestor/decoder.go), which is unaffected. Delete it from cmd/server. [T] grep on merged tree
F4 Nit TestOpenAPIDescriptionsHaveRoutes is a general OpenAPI invariant in an issue-specific file (post_packets_removed_223_test.go). It would be easier to find in openapi_test.go. Also, the author's log says the orphan check was added "in openapi.go". It is a test; openapi.go only loses the one description line. See also item 4. [A], [K]
F5 Nit, pre-existing The count-based exception for ping_score_history.go (15) allows a swap. Replace one of its own literals with an INSERT into another shared table and no guard fails (mutant M8 below survived). This PR did not cause it. A possible follow-up is to pin that file's write literals to its own tables (_meta, ping_score_history_entries). [T] mutant

1. Only POST /api/packets is removed

  • Routes: The diff removes exactly one registration (routes.go:351 on master) and its handler. [A]
  • Live probe: I built the real server binary for master and for the merged tree. Each ran against a migrated copy of test-fixtures/e2e-fixture.db, opened by OpenDB (mode=ro), using a throwaway local test key. [T]
Request master merged
POST /api/packets + valid key 500 {"error":"transmission insert: attempt to write a readonly database (8)"} 200 text/html (index.html, SPA catch-all)
POST /api/packets no key 401 200 text/html
GET /api/packets?limit=5, ?groupByHash=true, /api/packets/timestamps, GET /api/packets/{hash}, POST /api/packets/observations, POST /api/decode 200 200, byte-identical bodies
GET /api/spec — identical except that ('/api/packets','post') is gone
DB file sha256 / row counts (tx/obs/observers) unchanged, 499/500/31 unchanged, 499/500/31
  • What the PR pins, and on which router: [A]
    • TestPostPacketsRemovedReturns405OnReadOnlyDB uses the plain API router (RegisterRoutes only) and expects 405.
    • TestPostPacketsRemovedFallsThroughToSPAInProductionRouter adds the same PathPrefix("/") + wsOrStatic(spaHandler) wiring as main.go and expects 200 text/html.
    • Both use an OpenDB (mode=ro) store and check row counts. The PR text says plainly that production returns 200 index.html, not 405, and calls that the existing fallback. My live probe confirms it. [T]
    • The fallback for unmatched /api/* is tracked as fix(server): unknown /api/* paths and wrong methods return 200 index.html instead of a JSON 404/405 #233 and is out of scope here. [K]
  • Route survival: TestPacketsRoutesSurviveRemoval checks routing only, not handler output, and says so. My probe above covers the handler output. [A]/[T]

2. Everything that belonged only to the endpoint is gone

  • Removed:

    • handler and route;
    • PacketIngestResponse (types.go);
    • the OpenAPI description;
    • the packetpath import in routes.go (it is still used elsewhere);
    • TestPostPacketPersistsV3Schema;
    • PacketIngestRequest/PacketIngestResponse from proto/packet.proto, and the comment in proto/decoded.proto;
    • tools/generate-packets.js;
    • routes.go from knownServerWriteSQL.

    [A]

  • Leftovers I grepped for:

    • Grep for handlePostPacket, PacketIngest, generate-packets, "manual injection" and POST-to-/api/packets shapes across the repo, excluding firmware/ and node_modules. The only hits are the new tests, the new comment and the stale master comment (F2). [T]
    • PayloadJSON is left over (F3). ComputeContentHash (used by hash_migrate.go) and setupTestServerWithAPIKey (still used by tests) are still live. [T]
    • package.json does not reference the deleted tool. [T]
  • Proto:

    • Correct: the removed messages were referenced nowhere else. DecodedResult is still used by DecodeResponse and WSPacketData. [A]
    • Validator: tools/validate-protos.py gives the same result on master and merged: 38 fixtures, 0 errors, 3 warnings. The only change is 145 → 143 parsed messages. [T]
    • Necessary and harmless: the repo has no codegen and no generated .pb.go; the .proto files are documentation. Whole messages are removed, so no field numbers need reserved. An external client built from these messages could never have worked against a production (mode=ro) server, so nothing working can break. [A]
    • External clients: I found no MeshViewLive or other client reference in this repo, and I cannot check consumers outside it. [A]

3. Guard and the mode=ro probe

  • The new guard: TestServerHasNoPacketTableInserts covers INSERT, INSERT OR … and REPLACE on transmissions, observations, observers and dropped_packets. It runs on every non-test cmd/server/*.go with no exceptions, which is minimal. cmd/server has no Go subpackages; only testdata/. Its companion sensitivity test covers the three removed statements, plus quoted and multi-line forms. [A]/[T]
  • Generic INSERT guard: INSERTs into any table are still guarded by TestServerSourceHasNoNewWriteSQL. On the merged tree it allows only backup.go: 1 (VACUUM INTO), openapi.go: 1 (prose) and ping_score_history.go: 15 (its own DB). routes.go and hash_migrate.go are both at 0 now. [T]
  • Remaining write SQL:
    • No write-SQL literal is left outside those exceptions. [T]
    • The non-test Exec/Begin calls are: backup.go (VACUUM INTO), ping_score_history.go (own DB), reach_rank.go:349 (read-only tx, rollback only) and db.go:248 (PRAGMA wal_checkpoint on Close). The last two are pre-existing and out of scope. [A]
  • mode=ro probe repeated: see the table in item 1. Master gives 500 with SQLite text; merged gives no DB access and an unchanged file hash. [T]

4. Orphan check (TestOpenAPIDescriptionsHaveRoutes)

  • Robust. It builds METHOD path-template keys from router.Walk, exactly as buildOpenAPISpec looks descriptions up. So it can only flag a description that the spec builder would silently ignore anyway. [A]
  • RegisterRoutes has no conditional registration (97 .Methods(...) calls, no if/for), so config cannot hide a route from the test. [A]
  • Edge case: a description for a route registered outside RegisterRoutes (in main.go) would fail. Today that is only /ws, which is outside /api/ and has no description. Acceptable. [A]
  • The check is a test, not code in openapi.go (F4). Mutant M7 shows it works. [T]

5. Docs

  • Updated:

    • docs/api-spec.md: the section and its TOC entry are both gone.
    • docs/user-guide/faq.md Q8: now says MQTT + ingestor only, and that the server is read-only.
    • BUILD_PLAN.md: the injection line, the "manual packet injection" data line and the tool in the tree listing are removed.

    [A]

  • Other mentions: a grep of all *.md, *.json and *.yml finds no other "POST /api/packets" or injection mention. config.example.json and configuration.md describe apiKey generically ("POST/PUT routes"), which is still accurate. [T]

  • CHANGELOG: not touched. The reason is in the PR text. [K]

6. Diff contains only intended files

  • Files: git diff --stat origin/master...dbc04260 shows 13 files (+256/−563): one added (the new test), one deleted (tools/generate-packets.js), 11 modified. Every file matches the PR description. [T]

  • Hygiene:

    • one commit on top of 7697a826;
    • git diff --check is clean;
    • no conflict markers, no empty blobs in the tree, no mode changes;
    • the touched Go files are gofmt-clean.

    [T]

  • No trace of the git stash episode. [K] for the episode itself, [T] for the clean diff.

7. Rules

  • cmd/server read-only: the PR removes the last INSERTs on the shared DB. [T]
  • No new map[string]interface{}: there are none. The diff removes two, one in the handler and one in the deleted test. [T]
  • Fork guards: github.repository == appears 9 times in deploy.yml and once in release-fast-path.yml, unchanged. [T]
  • No closing keywords in the title, body or commit message; they use "Relates to POST /api/packets INSERTs on the server's read-only handle and always returns 500 #223". [T]
  • Commit author and committer: both are dborup <kontakt@meshview.dk>. [T]
  • Test count: top-level func Test in cmd/server goes 2114 → 2120 by my count (base vs head) and 2135 → 2141 on master vs merged. The delta of +6 (−1 / +7) matches the PR text; the PR's absolute numbers differ by 1 because of the counting method. [T]

Tests

Mutants (merged tree, targeted tests)

# Mutant Result
M1 Route + handler restored verbatim from master killed by 405 test (500), SPA test (500, JSON), TestOpenAPISpecHasNoPostPackets, TestServerHasNoPacketTableInserts, TestServerSourceHasNoNewWriteSQL (routes.go 3 > 0)
M2 One INSERT OR IGNORE INTO observers in routes.go killed by TestServerHasNoPacketTableInserts and TestServerSourceHasNoNewWriteSQL
M3 INSERT INTO nodes in routes.go (not a packet table) killed by TestServerSourceHasNoNewWriteSQL and TestServerSourceHasNoCachedRWCalls
M4 New guard widened (name == "routes.go" skipped) + INSERT INTO transmissions in routes.go killed by TestServerSourceHasNoNewWriteSQL; the generic guard backs up the widened one
M5 M4 + knownServerWriteSQL["routes.go"] = 1 (both guards widened) survived, as expected: it needs visible edits to two guard files
M6 Swap in exempt ping_score_history.go: a CREATE INDEX literal becomes INSERT INTO nodes (count still 15) killed by TestServerSourceHasNoCachedRWCalls
M7 "POST /api/packets" description re-added to routeDescriptions() killed by TestOpenAPIDescriptionsHaveRoutes
M8 Like M6, but INSERT INTO observer_neighbors survived: pre-existing gap (F5), not caused by this PR
M9 /api/packets GET route widened to Methods("GET","POST") killed by 405 test (200 JSON), SPA test, TestOpenAPISpecHasNoPostPackets

Not verified

  • CI on the merged tree: it needs the F1 sync first. My merged-tree results use my own conflict resolution.
  • Live UI: no browser check, no staging or prod. The probe ran only against local binaries and a fixture copy with a throwaway local test key.
  • External proto consumers: none in this repo, and I could not inspect any outside it.
  • The author's own runs: the 30m -race run and the proto-validator run, and fix(server): unknown /api/* paths and wrong methods return 200 index.html instead of a JSON 404/405 #233's scope. [K]
  • The cmd/ingestor test suite: not run, since this PR does not touch it.

dborup and others added 2 commits October 5, 2026 09:16
…ove-post-packets

Brings in #222 (content-hash migration moved to the ingestor), which also
rewrote the knownServerWriteSQL comment. The map merged cleanly to
backup.go 1, openapi.go 1, ping_score_history.go 15; only the comment
conflicted. Both notes are kept: hash_migrate.go lost its 3 literals in
#215 and routes.go lost its 3 with POST /api/packets (#223).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… review F2-F4)

F2: TestServerHasNoPacketTableWrites (#215) still described
handlePostPacket's INSERTs as a known gap, and #223 had added a second
guard, TestServerHasNoPacketTableInserts, with a different table list.
They are now one guard: INSERT joins UPDATE/DELETE/REPLACE in the
everywhere list, with one table list (transmissions, observations,
observers, dropped_packets, ping_triggers, route_mask_changes), no
exceptions, and every hit reported with its line. The INSERT cases and
the negative cases from the removed test move into
TestServerHasNoPacketTableWritesIsSensitive.

F3: PayloadJSON in cmd/server/decoder.go lost its only caller with the
POST handler. The ingestor's copy stays.

F4: TestOpenAPIDescriptionsHaveRoutes moves to openapi_test.go and uses
setupTestServer.

No behaviour change.

Relates to #223

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@adminopenclaw8-sketch

Copy link
Copy Markdown
Collaborator

Rapport — CS-Macmini PR#231 runde 2 — head 558e46f

Status: F1–F4 from the round-1 review are fixed; F5 has no change, and a follow-up issue is proposed below. CI is green (Go, Playwright, Docker). The PR stays draft and is ready for re-review by another reviewer.

Merged master (commit ab5ee85a)

Review feedback addressed (commit 558e46f2)

I took over from the original author (the cloud agent is no longer used) and pushed two fast-forward commits on top of dbc04260: no rebase, amend or force-push. Both commits are authored and committed by dborup <kontakt@meshview.dk>. [T]

Evidence tags: [T] = tested/run by me, [A] = read/analysed in source, [K] = taken from someone else's statement, not verified by me.

F1 — merge conflict with master (fixed, ab5ee85a)

F2 — one packet-table guard (fixed, 558e46f2)

F3 — dead PayloadJSON (fixed, 558e46f2)

  • PayloadJSON and the now-unused encoding/json import are removed from cmd/server/decoder.go. [T]
  • cmd/server has no references left (grep). cmd/ingestor/decoder.go keeps its own PayloadJSON, which cmd/ingestor/db.go still uses. [T]

F4 — orphan-description test location (fixed, 558e46f2)

F5 — ping_score_history.go count-based exception (no change, pre-existing)

  • No code change. I propose a follow-up issue (text below); I have not filed it. [T]
  • What changed for packet tables: a packet-table INSERT swapped into ping_score_history.go is now caught by the single guard (mutant R6 below). The remaining gap only concerns other shared tables, such as observer_neighbors. [T]

Tests

Run on the exact pushed tree (git archive 558e46f2) unless noted:

Mutants (copies of the 558e46f2 tree, targeted tests)

# Mutant Result
R1 INSERT OR IGNORE INTO observers re-added in routes.go killed by TestServerHasNoPacketTableWrites (routes.go:4822: INSERT OR IGNORE INTO observers) and TestServerSourceHasNoNewWriteSQL
R2 Route + handlePostPacket + PacketIngestResponse + PayloadJSON restored from master killed: TestPostPacketsRemovedReturns405OnReadOnlyDB (500 with "attempt to write a readonly database"), TestPostPacketsRemovedFallsThroughToSPAInProductionRouter, TestOpenAPISpecHasNoPostPackets, TestServerHasNoPacketTableWrites (all 3 INSERTs, with lines), TestServerSourceHasNoNewWriteSQL
R3 "POST /api/packets" description re-added to routeDescriptions() killed by TestOpenAPIDescriptionsHaveRoutes (in openapi_test.go)
R4 UPDATE observers SET … literal in store.go (newly covered table for UPDATE) killed by TestServerHasNoPacketTableWrites and TestServerSourceHasNoNewWriteSQL
R5 INSERT INTO dropped_packets in backup.go (a documented exception file) killed by TestServerHasNoPacketTableWrites and TestServerSourceHasNoNewWriteSQL (backup.go 2 > 1)
R6 Swap in exempt ping_score_history.go: one CREATE INDEX literal becomes INSERT INTO transmissions (count stays 15) killed by TestServerHasNoPacketTableWrites (ping_score_history.go:624)

CI (run 37277888063, head 558e46f2)

Job Result Duration
✅ Go Build & Test pass 22m58s
🎭 Playwright E2E Tests pass 20m04s
🏗️ Build & Publish Docker Image pass 53s
📦 Release Artifacts skipped (PR) —
🚀 Deploy Staging skipped (PR) —
📝 Publish Badges & Summary skipped (PR) —

[T] (gh run view, gh pr checks)

Proposed follow-up issue for F5 (not filed)

Title: test(server): pin ping_score_history.go's write SQL to its own tables

Body:

TestServerSourceHasNoNewWriteSQL allows ping_score_history.go 15 write-SQL literals, because that file writes its own history database, not the shared one. The allowance is a count, so replacing one of those literals with a write to a shared table keeps the count at 15 and passes.

This was found as a surviving mutant in the #231 review.

Proposal: for ping_score_history.go, require that every write-SQL literal names only that file's own tables (_meta, ping_score_history_entries and its indexes). Alternatively, check every write literal's target table against an allow-list per exception file. Add a sensitivity test that a swapped literal fails.

Acceptance:

Leftovers / not done

  • fix(server): unknown /api/* paths and wrong methods return 200 index.html instead of a JSON 404/405 #233: POST /api/packets, like any unmatched /api/* request, gets 200 index.html from the production router. That is out of scope here, and the current behaviour is pinned by TestPostPacketsRemovedFallsThroughToSPAInProductionRouter. [A]
  • cmd/server/decoder.go has pre-existing gofmt drift (see Tests). I did not touch it. [T]
  • Not re-checked this round: the round-1 end-to-end binary probe (master 500 vs merged 200 text/html, DB unchanged). Round 2 does not change any handler or route. [A]
  • Not done: no browser check, no staging or prod access, and the cmd/ingestor suite was not run, since round 2 does not touch it.
  • Next step: re-review by a different reviewer. I did not review my own fixes.

@dborup

dborup commented Oct 5, 2026

Copy link
Copy Markdown
Owner Author

Review — CS-MacBook PR#231 runde 2 — head 558e46f

Dom: APPROVE med nits

Evidence tags: [T] run or test output, [A] assessment or inference, [K] checked in code, diff, git or CI.

Setup: git fetch --no-prune origin; the head was 558e46f2 before and after (git ls-remote). Master is 3878d7ea, which already contains #234. The merged tree is git merge-tree --write-tree 3878d7ea 558e46f2: it merges cleanly (tree c25d9858). Everything below ran from git archive copies in scratch; no checkout was modified. The PR is still a draft and GitHub shows it as mergeable. [K]

Round-1 findings

# Round 1 Status Evidence
F1 Conflict with master in readonly_sql_literal_test.go Fixed in merge commit ab5ee85a (parents dbc04260, aff158c7). knownServerWriteSQL is backup.go: 1, openapi.go: 1, ping_score_history.go: 15; neither hash_migrate.go nor routes.go is listed. The two history notes are merged into one comment. The merge commit changes the same 13 files as the original commit (+257/−564 against +256/−563: the one-line difference is the merged comment). [K]
F2 Two packet-table guards with different table lists; stale "known gap" comment Fixed in 558e46f2. See point 2 below. [K][T]
F3 Dead PayloadJSON in cmd/server Fixed. Deleted, with the now-unused encoding/json import. cmd/server has no reference left. cmd/ingestor/decoder.go keeps its copy (still used by cmd/ingestor/db.go), and git diff of cmd/ingestor between master and head is empty. [K][T]
F4 Orphan-OpenAPI test in an issue-specific file Fixed. TestOpenAPIDescriptionsHaveRoutes is in openapi_test.go (line 148), using the shared setupTestServer. The header of post_packets_removed_223_test.go points at the single guard. [K]
F5 The count-based exception for ping_score_history.go allows a swap Open, pre-existing, not a blocker. A packet-table swap there is now caught (mutant R6 in the author's report; I did not rerun R6). A swap to a non-packet shared table such as observer_neighbors still survives. Follow-up proposed, not filed. [K]

New findings

# Sev Finding
N1 nit, information The guard is textual, so it is a tripwire and not a proof. Mutant X4 (below) splits the SQL across literals, "UPDATE " + "observers" + " SET name = 'x'", and both guards pass. A schema-qualified INSERT INTO main.transmissions is missed by the packet guard but still caught by the generic literal guard. Neither is a regression; I mention it so nobody reads the "no exceptions" wording as "unbypassable". [T]
N2 information The packet-table list is the six tables named in the task. The ingestor also owns client_observers, client_receptions, obs_mask_seen and observer_metrics. A write to those from the server is still caught by the generic count guard, but not by the packet guard. Not a defect; the list is what was asked for. [K][A]

No blocking finding.

1. F1 — merge and knownServerWriteSQL

2. F2 — single guard

  • Coverage. TestServerHasNoPacketTableWrites (readonly_invariant_test.go) uses packetTableWritePatterns(): INSERT / INSERT OR …, UPDATE / UPDATE OR …, DELETE FROM, REPLACE INTO. They are all built from one txTableWritePattern, whose one table list is transmissions, observations, observers, dropped_packets, ping_triggers, route_mask_changes. [K]
  • Scope. It reads every non-test *.go in cmd/server with no exceptions. cmd/server has no Go subpackages (find returns none). It matches raw text, including comments, so a comment quoting such a statement would also fail; today none do. The hash_migrate*.go files additionally may not call .conn.Exec/Begin/Prepare…; hash_migrate.go still exists in cmd/server. [K][T]
  • DRY. One guard, one table list, one pattern builder. TestServerHasNoPacketTableWritesIsSensitive uses the same packetTableWritePatterns(). It checks positive cases (the old migration statements, the removed handler's INSERTs, a multi-line quoted form, REPLACE on dropped_packets, DELETE on route_mask_changes) and negative cases (observer_neighbors, observations_archive, a SELECT, tx_inserted). The removed TestServerHasNoPacketTableInserts and its pattern are gone from post_packets_removed_223_test.go. [K]
  • Reporting. Violations now list every hit with file, line and matched text (FindAllIndex), not only the first per pattern. [K]
  • Comment. The new text is accurate: the history of hash_migrate: server-side DB writes on the read-only handle; failures logged as collisions; ghost duplicates share a hash #215 and POST /api/packets INSERTs on the server's read-only handle and always returns 500 #223; the rule "INSERT, UPDATE, DELETE and REPLACE on these tables are forbidden in every server source file"; and the claim that the documented exceptions (ping_score_history.go, backup.go) do not touch them. All three hold on the merged tree, and the guard passes. The one caveat is N1: the guard is textual. [K][T]
  • Count of tests (top-level func Test in cmd/server): 2135 on master, 2139 on the merged tree, as the author reported. [T]

3. Behaviour of the removal

Built the real server for master (3878d7ea) and for the merged tree. Each ran against a copy of test-fixtures/e2e-fixture.db migrated with cmd/migrate (the CI step), with a throwaway local key written to a local config file, on localhost ports. Servers were stopped by port. [T]

Request master merged
POST /api/packets with key 500 application/json (SQLite read-only error) 200 text/html (SPA index)
POST /api/packets without key 401 200 text/html
GET /api/packets?limit=5 200 200, body identical
GET /api/packets?groupByHash=true&limit=5 200 200, body identical
GET /api/packets/timestamps (no params) 400 400, body identical
GET /api/packets/{hash}, GET /api/packets/{hash}/path 200 200, bodies identical
POST /api/decode 200 200, body identical
POST /api/packets/observations 200 200, body identical
GET /api/spec has POST /api/packets the same, minus that one operation (every other operation, info, components and tags identical)

The DB file hash is the same before and after both runs. The 200 text/html for the removed route is the existing SPA fallback, covered by TestPostPacketsRemovedFallsThroughToSPAInProductionRouter and tracked as #233. [T]

The diff removes exactly one route registration and its handler; the 405 behaviour on the plain API router is pinned by TestPostPacketsRemovedReturns405OnReadOnlyDB. [K]

4. Rules

  • Fork guards: deploy.yml 9, release-fast-path.yml 1 (github.repository == 'Kpa-clawbot/CoreScope'); no workflow file changed. [K]
  • New map[string]interface{}: 0 added, 2 removed. [K]
  • No closing keywords in the PR body or any commit message. [K]
  • All three commits: author and committer dborup <kontakt@meshview.dk>. [K]
  • cmd/server read-only: the PR removes the last INSERTs on the shared DB. [K]
  • gofmt -l: clean on the touched files except decoder.go, which is already listed on master (struct-tag alignment, outside the changed lines). go vet is clean. [T]

Tests

  • cd cmd/server && go test -race -count=1 -timeout 30m ./... on the merged tree (master 3878d7ea + head): ok, 395.3 s, exit 0, 0 DATA RACE lines. None of the known flakes fired. [T]
  • sh test-all.sh on the merged tree: 217 passed, 0 failed (217 files). [T]
  • CI on the head: Go Build & Test, Playwright E2E and Docker pass; Release, Deploy and Badges are skipped (PR). It ran on the earlier base, not on current master. [K]

Mutants (copies of the merged tree; targeted tests)

# Mutant Result
X1 INSERT INTO transmissions literal in routes.go killed by TestServerHasNoPacketTableWrites (routes.go:4822: INSERT INTO transmissions) and TestServerSourceHasNoNewWriteSQL
X2 UPDATE observations SET … in neighbor_api.go (a random server file) killed by both (neighbor_api.go:571)
X3 UPDATE observers SET … in backup.go (a documented exception file) killed by both
X4 Split literal "UPDATE " + "observers" + " SET …" in neighbor_api.go survived: see N1
X5 INSERT INTO main.transmissions in neighbor_api.go killed by TestServerSourceHasNoNewWriteSQL only; the packet guard misses the schema prefix (N1)
X6 POST /api/packets route re-registered with a stub handler killed by the 405 test, the SPA-fallback test and TestOpenAPISpecHasNoPostPackets

Not verified

  • CI on the merged tree. My merged tree has no CI run; I relied on my own local runs.
  • No browser check and no staging or prod. The probe ran only against local binaries and a fixture copy.
  • External consumers of the removed proto messages, outside this repo.
  • The cmd/ingestor suite: not run, since the PR does not touch it (the diff of cmd/ingestor is empty).
  • The author's own mutants R1–R6: I checked the guard on X1–X3 myself and did not repeat R5 or R6.
  • Whether the F5 follow-up issue exists: the report says it was not filed.

@dborup
dborup marked this pull request as ready for review October 5, 2026 08:41
@dborup
dborup merged commit 2a7877a into master Oct 5, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants