analytics(hash-stats): break down path hash sizes by payload type and channel

[dborup]

Summary

Extend Analytics → Hash Stats with a breakdown of the 1/2/3-byte path hash distribution per payload type (and, for channel messages, per channel), so it is visible how far each kind of traffic has moved to multi-byte hashes (e.g. chat vs. adverts).

Background

• computeAnalyticsHashSizes (cmd/server/store.go, around line 8313) already reads the path byte (bits 7–6 of the path-length byte in raw_hex) for every transmission and returns total, distribution, distributionByRepeaters, hourly, topHops and multiByteNodes, with region/area filtering.
• feat(channels): show observed path hash sizes #131 added an observation-based width for channel messages only. It is in-memory only, GRP_TXT only, gives nothing for zero-hop floods, and "mixed" widths did not occur on staging (0 of 516 messages). It is therefore not a good analytics source; this issue uses the path byte, which Hash Stats already parses and which also covers zero-hop floods (see the firmware notes in Kpa-clawbot/CoreScope#2089).

Proposal

Server (same loop, no new query, no schema change):

• Count distribution per payload type: {payloadType → {1,2,3}} using the firmware type names already used elsewhere (ADVERT, GRP_TXT, TXT_MSG, REQ, …).
• For GRP_TXT, count per channel key: the stored channel_hash (decrypted name such as #name, or enc_XX when not decrypted). Keep only the top N channels by message count (N = 20 initially) and fold the rest into other, so the result stays bounded.
• Respect the existing region/area filters and caching/recompute behaviour of the Hash Stats endpoint.
• Add the new fields as named Go structs with explicit JSON tags (AGENTS.md: no new map[string]interface{} in response builders), e.g. distributionByPayloadType and grpTxtByChannel. Existing fields stay unchanged.

Frontend (public/analytics.js, Hash Stats tab):

• A compact table (or stacked bars) with one row per payload type: count and share of 1/2/3-byte.
• A second table for the top channels, with the same columns.
• Colours via existing CSS variables/badges (badge-hash-N); all names escaped.
• Deep-linkable section (#/analytics?tab=hashsizes&section=…).

Acceptance criteria

• Per-type and per-channel counts sum to the overall distribution for the same filter (test).
• Unknown/reserved size bits and packets without a path byte are excluded exactly as today.
• Region and area filters apply to the new fields (test).
• Per-channel map is bounded (top N + other); N is a named constant, noted for the customizer later (AGENTS.md rule 8).
• Work stays O(1) extra per transmission in the existing loop; include a benchmark or timing for the recompute on a realistic store (AGENTS.md rule 0).
• Unit tests for the server aggregation and the frontend rendering (including escaping of channel names); Playwright check of the tab at desktop and mobile width.
• docs/api-spec.md updated.

Non-goals

• No use of the feat(channels): show observed path hash sizes #131 observation mask.
• No per-sender statistics (channel senders are only known by decrypted name).
• No new endpoint, schema change or deploy in this issue.

[dborup]

Specification review

Reviewed against master at 5f493f1de6bbf1eff4078790476c72667f608a74 and the issue description last updated on 2026-09-29 at 11:18 UTC.

The feature is useful, but the measurement contract needs clarification before implementation. These are findings about the proposal and the existing behavior it would inherit, not a review of an implementation PR.

1. [P1] The first retained raw frame is not necessarily representative of the transmission

The aggregate reads the width from tx.RawHex, while a deduplicated transmission can contain observations from different routes. StoreTx explicitly documents that the first-ingested frame is not a stable description of all observed routes.

For example, a transmission first retained as a direct/zero-hop frame with path byte 0x00 can later acquire a non-empty 2-byte flood path. The current aggregate can then pass the non-empty-path check using the later observation, but classify the width as 1 byte using the retained raw header. The header and the relay-path evidence are describing different observations.

Define which frame/evidence determines the new distribution, and how unknown or conflicting evidence is handled. Add a regression covering direct-first/flood-later and the reverse ingestion order. The reported absence of mixed widths in a 516-message staging sample does not establish a protocol invariant or resolve this case. This does not require adopting the #131 mask unchanged, but the alternative needs an explicit correctness contract.

Sources: StoreTx evidence model, raw-header width calculation.

2. [P2] Including zero-hop floods conflicts with preserving the existing distribution totals

The existing distribution and total exclude transmissions whose selected path is empty. Parsing a width from their headers does not mean these transmissions currently contribute to those totals.

If the new tables include zero-hop floods, they cannot also reconcile to the unchanged legacy distribution. Choose either:

• the same eligible population as the existing distribution; or
• a separately named total/distribution for the expanded population, with explicit reconciliation rules.

Source: non-empty-path gate before total/distribution updates.

3. [P2] Channel counts must reconcile to GRP_TXT, not to all payload types

The first acceptance criterion currently requires both per-type and per-channel counts to sum to the overall distribution. Channel rows cover only GRP_TXT, so this cannot hold when eligible ADVERT or other traffic is present.

Use these invariants separately for each width:

• All payload-type rows sum to the selected overall distribution, including an explicit unknown-type policy.
• Channel rows, including other and any missing-key/unknown bucket, sum to the GRP_TXT row.

Also specify whether each percentage is relative to its row total or the overall population.

4. [P2] Reusing the existing area filter does not filter channel traffic

Inside computeAnalyticsHashSizes, the area check only excludes parsed ADVERT originators outside the area. It does not restrict GRP_TXT transmissions.

Define what channel traffic belongs to an area, or explicitly expose/document that this breakdown does not support that area interpretation. Merely placing new counters after the current filter does not satisfy the proposed area-filter acceptance criterion.

Source: ADVERT-only area check.

5. [P2] The stored channel_hash is not currently available in this loop

transmissions.channel_hash exists in SQLite, but StoreTx has no corresponding field and the current load query does not select it.

Specify whether to carry the stored value through the existing loading/live-update paths or derive an equivalent key from decoded data. The latter needs consistent normalization, missing-value behavior, and tests against the ingestor mapping. This is compatible with avoiding a new query, but it is not just an extra counter in the current loop.

Sources: StoreTx, load query, ingestor channel-key mapping.

Additional acceptance-test and resource-contract details

• Explicitly cover TRACE, whose header path is repurposed for SNR rather than normal hop hashes; zero-hop flood versus direct; transport-header offsets; declared 2/3-byte zero-hop cases; reserved bits; and malformed/truncated input. Do not silently turn an absent size declaration into 1-byte evidence. See the existing PathBytesAreHops helper and firmware behavior.
• Top N + other bounds the response, not automatically the intermediate channel-count map. Specify the accumulator bound and deterministic tie-breaking, while preserving exact counts. Include that allocation cost in the benchmark.

Recommendation: settle the eligible population, width-selection rule, channel identity, filter semantics, and reconciliation invariants first, then implement the tables.

Read-only specification/code review; no code changes or test execution were performed.