Repository navigation
observer: filter enhancements + MQTT fields (advert window, path block, dry-run, rules, message age) - #12
Merged
Elektr0Vodka merged 14 commits intoSep 26, 2026
Conversation
DMC_CHANNEL=dev stamps a -dev marker into the embedded version (before the fork tag) and points OTA_MANIFEST_BASE_URL at the dev channel (/mqtt/dev/v); anything else stays on stable (/mqtt/v). Reconciled with the stable/dev base bake so both live in one build.sh. An explicit OTA_MANIFEST_BASE_URL still wins.
…irtime Ports the four ideas from the jhuebert/MeshCore repeater-filter fork that fit our per-type filter without adopting its rule engine or grammar: - `filter advert <hours>`: each origin's flood advert is forwarded at most once per window (0-720 h, 256-entry ring keyed on 4 pubkey bytes), ahead of the per-type advert limiter so repeats never eat the legit budget. - `filter path add|remove|list <hex>`: drop flood packets whose path holds a repeater ID starting with one of up to 8 prefixes (2-8 hex digits), compared aligned to the packet's hash size. - `filter dryrun on|off`: every drop is counted but still forwarded; the status line gains a trailing `(dry-run)` marker after the bracket. - `filter stats air`: estimated time-on-air the drops saved, via the radio's own estimate, billed only on drops. New stats topics `advert`, `path`, `air`; existing replies are unchanged except the condensed `filter help` line. New prefs are appended to /filter_prefs and defaulted per field when an older file is shorter. Header-only AdvertLimiter/PathBlock plus the FilterStats additions are covered by native tests (test_filter_policy, test_filterstats).
…r-fork # Conflicts: # .gitignore
… contract The `filter` topic now carries the fork-derived filter additions: `dryrun`, `totals.advert` / `totals.path`, `air_ms`, an `advert` block (window_h, cache, cache_size) and a `paths` array (prefix, drops; omitted when empty). The JSON shaping moves out of MQTTMessageBuilder.cpp (which needs Timezone and Mesh headers and cannot build on native) into the header-only MQTTFilterStatsJson.h over a dependency-free MQTTFilterStatsView.h, so the payload contract is covered by test_mqtt_filter_stats_json. That test also bounds a heavy realistic payload (all types dropping, 4 channels, 8 top sources, 8 path prefixes) at ~2.4 KB, which no longer fit the 2 KB publish buffer; FILTER_JSON_BUFFER_SIZE grows to 3 KB (two static buffers, ESP32 observer targets only).
Adapts the remaining jhuebert-fork ideas into our grammar without its rule engine or regex: - `filter sender add <name> [secs] [prob]` / remove / list: exact name or `Prefix*`; secs=0 blocks, otherwise one match per secs passes and the excess is dropped; prob 1-100 is the share of matches the rule decides. - `filter text add <pattern> [secs] [prob]`: substring, or `^prefix`. - `filter watch add|remove|list <#name>`: up to 4 channels whose key is derived from the name and decrypted for the rules; Public is always read. - `filter stats sender|text`: drops per rule plus throttle passes. Rules are evaluated top to bottom (senders, then texts); a failed roll or a within-budget throttle pass steps aside so a later rule can still decide. Only plain group texts (`Sender: text`) match. The one decrypt per packet is shared with the malformed scan and only happens when a rule exists. Header-only SenderRules.h (parse, match, evaluate, arg parsing) and the FilterStats formatter are covered by test_filter_rules and test_filterstats. Prefs append 8+8 rules and 4 watch channels behind the earlier fields with the same per-field load defaults.
…erver-dev-filtering
The `filter` topic gains `totals.sender` / `totals.text`, `senders[]` and `texts[]` (pattern, secs, prob, drops, pass in evaluation order) and `watch[]`, each omitted when empty. The heavy realistic payload now reaches ~3.4 KB, so FILTER_JSON_BUFFER_SIZE grows to 4 KB; the bound stays asserted by test_mqtt_filter_stats_json.
8efae0e) Heltec V4, Heltec V4 r8, Heltec Wireless Tracker and Xiao S3 WIO repeater observer_mqtt bins (app + merged + partsig), built locally on the dev channel so testers can flash without a release. See prebuilts/README.md.
…341f1) Heltec V3 (ESP32, app + merged) and RAK4631 (nRF52, uf2 + zip) repeater bins built locally as v1.17.1-dev so testers can flash without a release. See prebuilts/README.md.
`filter age <minutes>|off` drops group texts whose sender timestamp is older than the limit (1-10080 min, default off), on every channel the repeater can read: Public plus the watch list. Direct messages and other channels are encrypted with keys the repeater does not hold, so their age cannot be read. Drops are counted under `filter stats age`; the setting is appended to FilterPrefs and defaults to off when an older /filter_prefs is loaded. The check trusts the repeater clock, so it is inactive while that clock is unset (before 2026-01-01; an RTC-less repeater boots at 15 May 2024) and `filter age` then says so. The malformed scan's +-1 week window gets the same guard: before, a repeater with an unset clock dropped every current Public message as "time" malformed once the scan was on. Timestamp logic lives in header-only MessageAge.h with native tests.
The cert bundle, cacert.pem, generated WebConfigHtml.h and a .pyc are generated by observer builds and were left over in the worktree.
…ancement/dmc-observer-dev-filtering # Conflicts: # prebuilts/README.md
The `filter` message gains `totals.age` (group texts dropped by `filter age`) and an `age` block with `max_mins` (0 = off) and `clock_set` (false while the repeater clock is unset, when the limit is inactive). Covered by the payload contract test, including the heavy-payload size check.
…mc-observer-dev-filtering
Elektr0Vodka
added a commit
that referenced
this pull request
Sep 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Observer counterpart of #11: brings the repeater filter enhancements into
dmc-observer-devand publishes them in the MQTTfiltermessage.Firmware
Everything in #11 (dry-run, per-origin advert window, path-prefix block, sender/text rules, watch list,
filter agefor #8, and the malformed-scan clock guard), merged fromenhancement/dmc-dev-filtering.MQTT
filtermessagedryrun,air_mstotals.advert/path/sender/text/ageadvert { window_h, cache, cache_size }age { max_mins, clock_set }:clock_setis false while the limit is inactivepaths[],senders[],texts[],watch[], each omitted when emptyJSON shaping lives in header-only
src/helpers/MQTTFilterStatsJson.hover theMQTTFilterStatsView.hPOD, so the payload contract is unit-tested onnative(test_mqtt_filter_stats_json). A heavy-payload test checks the worst case still fits the publish buffer, which is now 4096 bytes (FILTER_JSON_BUFFER_SIZE).Testing
Heltec_v3_repeater_observer_mqttbuilds.Before merging
prebuilts/holds ~13 MB of observer test binaries (heltec_v4, heltec_v4_r8, Heltec Wireless Tracker, Xiao S3 WIO) committed for testers. Decide whether they should land ondmc-observer-devor be dropped first.7f52c992(DMC MkDocs site) and3bf3a2aa(OTA channel build coupling), which are not ondmc-observer-devyet.docs/cli_commands.md.🤖 Generated with Claude Code