Add malformed chat filtering and validation for public messages

- Introduced a new feature to validate and filter malformed chat messages in companion chat views.
- Implemented logic to drop malformed public chat messages at repeaters by default, ensuring binary datagrams remain unaffected.
- Enhanced text validation with UTF-8 checks and quality metrics to improve message display.
- Updated documentation to reflect changes in chat handling and configuration options for malformed message handling.
- Added new methods and structures to support the tracking and management of malformed message statistics.

Author: MichTronics

README-NL.md

@@ -4,6 +4,8 @@ MeshCoreNG is een Next Gen variant van MeshCore.
 
 Kort gezegd: MeshCore laat LoRa-apparaten berichten naar elkaar doorgeven zonder internet. MeshCoreNG bouwt daarop verder en probeert vooral repeaters slimmer te maken, zodat grotere en drukkere netwerken beter blijven werken.
 
+Website en webflasher: https://michtronics.github.io/MeshCoreNG/
+
 Het doel is niet om MeshCore opnieuw te bouwen. Het doel is om stap voor stap verbeteringen toe te voegen, zonder bestaande clients of het bestaande protocol kapot te maken.
 
 ## Waarom dit in Nederland belangrijk is
@@ -463,13 +465,30 @@ region tree
 region save
 ```
 
+**Malformed chat afhandeling:**
+
+Companion radio firmware valideert menselijke chat voordat die naar apps of displays gaat. Ongeldige UTF-8, binary-achtige tekst, te veel control characters, replacement characters, onmogelijke timestamps en tekst met een heel lage confidence score worden bij de chat/UI-laag gefilterd. Binary datagrams, raw/custom packets, requests, responses en toekomstige packet types blijven binary-safe.
+
+Standaard wordt malformed companion chat als compacte placeholder getoond, zodat rommeltekst niet in de Android/app kant gerenderd wordt. Payloads die niet geïnspecteerd kunnen worden, binary channel datagrams en onbekende/toekomstige packet types worden niet blind gedropt.
+
+Repeater firmware kan ook ingesteld worden om malformed default-public-channel group text te droppen voordat het opnieuw wordt uitgezonden:
+
+```text
+get malformed.drop
+set malformed.drop on
+set malformed.drop off
+```
+
+Dit staat standaard aan op repeaters. Repeaters droppen alleen tekstpackets die ze kunnen inspecteren en als malformed kunnen classificeren. Encrypted/private group text die de repeater niet kan decrypten, binary datagrams en onbekende/toekomstige packet types blijven volgens de normale forwardingregels lopen.
+
 Meer CLI-uitleg staat in [docs/cli_commands.md](./docs/cli_commands.md).
 
 ## Compatibiliteit
 
 MeshCoreNG blijft compatible met het bestaande MeshCore ecosysteem.
 
 - Geen packet format wijziging voor deze dense-mesh stappen.
+- Chat-sanitatie geldt alleen voor menselijke chatweergave/forwarding policy; binary transport blijft ondersteund.
 - Bestaande MeshCore clients blijven werken.
 - Bestaande MeshCore firmware kan nog steeds met MeshCoreNG praten.
 - De standaardinstellingen blijven veilig voor normale en sparse netwerken.
@@ -498,9 +517,12 @@ Voor developers:
 MeshCoreNG heeft nu een GitHub Pages webflasher voor ESP32 repeater builds:
 
 - MeshCoreNG webflasher: https://michtronics.github.io/MeshCoreNG/flasher/
+- MeshCoreNG website en docs: https://michtronics.github.io/MeshCoreNG/
 
 De flasher gebruikt ESP Web Tools en werkt vanuit Chrome of Edge met Web Serial. Hij is bedoeld voor ESP32-family boards. nRF52, RP2040 en STM32 boards gebruiken nog steeds hun normale firmwarebestanden en flashing tools.
 
+ESP32 repeater builds die je via deze site flasht hebben malformed public chat dropping standaard aan. Controleer of wijzig dit na het flashen met `get malformed.drop`, `set malformed.drop on` of `set malformed.drop off`.
+
 De firmwarebestanden die de webflasher gebruikt komen uit GitHub Release assets. De release/CI workflow bouwt de ESP32 repeater-varianten, hangt de merged `.bin` bestanden aan de release, en de GitHub Pages workflow downloadt daarna die releasebestanden om de ESP Web Tools manifests onder `/flasher/` te maken.
 
 Wil je later nog een ESP32-board toevoegen aan de webflasher, dan voeg je de PlatformIO environment name, display name, chip family en beschrijving toe aan `webflasher/boards.json`. De bijbehorende release asset moet een naam hebben zoals `<env>-*-merged.bin`.

README.md

@@ -4,6 +4,8 @@ MeshCoreNG is a Next Gen variant of MeshCore.
 
 In simple terms: MeshCore lets LoRa devices pass messages to each other without the internet. MeshCoreNG builds on that and focuses on making repeaters smarter, so larger and busier networks can keep working better.
 
+Website and web flasher: https://michtronics.github.io/MeshCoreNG/
+
 The goal is not to rebuild MeshCore from scratch. The goal is to add improvements step by step, without breaking existing clients or the existing protocol.
 
 ## Why This Matters In The Netherlands
@@ -466,13 +468,30 @@ region tree
 region save
 ```
 
+**Malformed chat handling:**
+
+Companion radio firmware validates human-readable chat before it is shown to apps or displays. Invalid UTF-8, binary-looking text, excessive control characters, replacement characters, impossible timestamps and very low-confidence text are filtered at the chat rendering boundary. Binary datagrams, raw/custom packets, requests, responses and future packet types remain binary-safe.
+
+By default malformed companion chat is shown as a compact filtered placeholder, so garbage text is not rendered in the Android/app side. Payloads that cannot be inspected, binary channel datagrams and unknown/future packet types are not blindly dropped.
+
+Repeater firmware can be configured to drop malformed default-public-channel group text before retransmission:
+
+```text
+get malformed.drop
+set malformed.drop on
+set malformed.drop off
+```
+
+This is enabled by default on repeaters. Repeaters only drop text packets they can inspect and classify as malformed. Encrypted/private group text that the repeater cannot decrypt, binary datagrams and unknown/future packet types are still relayed according to the normal forwarding rules.
+
 More CLI details are in [docs/cli_commands.md](./docs/cli_commands.md).
 
 ## Compatibility
 
 MeshCoreNG remains compatible with the existing MeshCore ecosystem.
 
 - No packet format change for these dense-mesh steps.
+- Chat sanitation is applied only to human-readable chat display/forwarding policy; binary transport stays supported.
 - Existing MeshCore clients keep working.
 - Existing MeshCore firmware can still talk to MeshCoreNG.
 - The default settings remain safe for normal and sparse networks.
@@ -501,9 +520,12 @@ For developers:
 MeshCoreNG now includes a GitHub Pages web flasher for ESP32-based repeater builds:
 
 - MeshCoreNG web flasher: https://michtronics.github.io/MeshCoreNG/flasher/
+- MeshCoreNG website and docs: https://michtronics.github.io/MeshCoreNG/
 
 The flasher is built with ESP Web Tools and works from Chrome or Edge using Web Serial. It is meant for ESP32-family boards. nRF52, RP2040 and STM32 boards still use their normal flashing files and tools.
 
+ESP32 repeater builds flashed from this site have malformed public chat dropping enabled by default. Check or change it after flashing with `get malformed.drop`, `set malformed.drop on`, or `set malformed.drop off`.
+
 The firmware files used by the web flasher come from GitHub Release assets. The release/CI workflow builds the ESP32 repeater variants, attaches the merged `.bin` files to the release, and the GitHub Pages workflow downloads those release files to create the ESP Web Tools manifests under `/flasher/`.
 
 To add another ESP32 board to the web flasher, add its PlatformIO environment name, display name, chip family, and description to `webflasher/boards.json`. The matching release asset must be named like `<env>-*-merged.bin`.

docs/cli_commands.md

@@ -474,6 +474,20 @@ On ESP32 boards with supported LoRa DIO1 wake wiring, sleep can wake by LoRa RX
 
 ---
 
+#### View or change malformed repeater forwarding
+**Usage:**
+- `get malformed.drop`
+- `set malformed.drop on`
+- `set malformed.drop off`
+
+**Parameters:**
+- `on`: drop malformed decryptable default-public-channel group text before retransmission
+- `off`: leave forwarding behavior unchanged
+
+**Default:** `on`
+
+**Note:** This only applies to human-readable default public group text that the repeater can decrypt and inspect. Binary channel datagrams, raw/custom payloads, requests, responses, private/encrypted packets that cannot be inspected and unknown/future packet types are still handled by the normal forwarding rules. Existing saved preferences are preserved; use `set malformed.drop off` to disable this behavior on a repeater.
+
 #### View or change this node's advert path hash size
 **Usage:**
 - `get path.hash.mode`

docs/companion_protocol.md

@@ -272,6 +272,8 @@ Bytes 7+: Message Text (UTF-8, variable length)
 
 **Timestamp**: Unix timestamp in seconds (32-bit unsigned integer, little-endian)
 
+**Text validity**: Channel text is expected to be valid human-readable UTF-8. Firmware validates received text before queuing it to the companion app. Malformed incoming text may be replaced with `[Malformed packet filtered]`. This does not affect channel data datagrams.
+
 **Example** (send "Hello" to channel 1 at timestamp 1234567890):
 ```
 03 00 01 D2 02 96 49 48 65 6C 6C 6F
@@ -329,6 +331,10 @@ Byte 0: 0x0A
 
 **Note**: Poll this command periodically to retrieve queued messages. The device may also send `PACKET_MESSAGES_WAITING` (0x83) as a notification when messages are available.
 
+**Malformed chat behavior**: Received contact and channel text is sanitized before it is queued. The firmware rejects malformed UTF-8, excessive control/replacement characters, binary-looking text and impossible timestamps. A malformed chat item may be queued as `[Malformed packet filtered]` instead of raw garbage text.
+
+Binary channel data received through `CMD_SEND_CHANNEL_DATA` / channel data callbacks is not sanitized as text and remains binary-safe.
+
 ---
 
 ### 7. Get Battery and Storage

docs/index.md

@@ -9,6 +9,7 @@ Below are a few quick start guides.
 - [Companion Protocol](./companion_protocol.md)
 - [Dutch Region Database](./dutch_region_db.md)
 - [Packet Format](./packet_format.md)
+- [Payloads](./payloads.md)
 - [QR Codes](./qr_codes.md)
 
 If you find a mistake in any of our documentation, or find something is missing, please feel free to open a pull request for us to review.

docs/payloads.md

@@ -174,6 +174,8 @@ txt_type
 | `0x01` | CLI command               | the command text of the message                            |
 | `0x02` | signed plain text message | first four bytes is sender pubkey prefix, followed by plain text message |
 
+Human-readable text payloads are validated before firmware UI/app callbacks render them. Implementations may reject or hide malformed UTF-8, excessive replacement/control characters, binary-looking high-entropy text and impossible timestamps. This validation is a display/forwarding policy for text payloads only; it does not change the packet format and does not apply to binary datagram, raw/custom, request, response or unknown/future payload types.
+
 # Anonymous request
 
 | Field            | Size (bytes)    | Description                               |
@@ -236,6 +238,8 @@ txt_type
 
 The plaintext contained in the ciphertext matches the format described in [plain text message](#plain-text-message). Specifically, it consists of a four byte timestamp, a flags byte, and the message. The flags byte will generally be `0x00` because it is a "plain text message". The message will be of the form `<sender name>: <message body>` (eg., `user123: I'm on my way`).
 
+Nodes that can decrypt a group text packet may apply the same human-readable text validation before displaying or forwarding it. Companion radio firmware sanitizes malformed decryptable group text before queuing it to the app and does not expose a separate malformed-drop command. Repeater firmware can be configured with `set malformed.drop on` to drop malformed decryptable default-public-channel group text before retransmission. Group datagrams remain binary payloads and are not subject to this text filter.
+
 # Group datagram
 
 | Field        | Size (bytes)    | Description                                |

examples/companion_radio/DataStore.cpp

@@ -233,6 +233,7 @@ void DataStore::loadPrefsInt(const char *filename, NodePrefs& _prefs, double& no
     file.read((uint8_t *)&_prefs.rx_boosted_gain, sizeof(_prefs.rx_boosted_gain));         // 89
     file.read((uint8_t *)_prefs.default_scope_name, sizeof(_prefs.default_scope_name));    // 90
     file.read((uint8_t *)_prefs.default_scope_key, sizeof(_prefs.default_scope_key));     // 121
+    file.read(pad, 1);                                                                     // 137 : reserved padding
 
     file.close();
   }
@@ -273,6 +274,7 @@ void DataStore::savePrefs(const NodePrefs& _prefs, double node_lat, double node_
     file.write((uint8_t *)&_prefs.rx_boosted_gain, sizeof(_prefs.rx_boosted_gain));         // 89
     file.write((uint8_t *)_prefs.default_scope_name, sizeof(_prefs.default_scope_name));    // 90
     file.write((uint8_t *)_prefs.default_scope_key, sizeof(_prefs.default_scope_key));     // 121
+    file.write(pad, 1);                                                                     // 137 : unused
 
     file.close();
   }

examples/companion_radio/MyMesh.cpp

@@ -530,6 +530,14 @@ void MyMesh::onSignedMessageRecv(const ContactInfo &from, mesh::Packet *pkt, uin
   queueMessage(from, TXT_TYPE_SIGNED_PLAIN, pkt, sender_timestamp, sender_prefix, 4, text);
 }
 
+void MyMesh::onMalformedMessageRecv(const ContactInfo &from, mesh::Packet *pkt, uint32_t sender_timestamp,
+                                    const char *reason, uint8_t score) {
+  (void)reason;
+  (void)score;
+  markConnectionActive(from);
+  queueMessage(from, TXT_TYPE_PLAIN, pkt, sender_timestamp, NULL, 0, TXT_MALFORMED_PLACEHOLDER);
+}
+
 void MyMesh::onChannelMessageRecv(const mesh::GroupChannel &channel, mesh::Packet *pkt, uint32_t timestamp,
                                   const char *text) {
   int i = 0;
@@ -577,6 +585,13 @@ void MyMesh::onChannelMessageRecv(const mesh::GroupChannel &channel, mesh::Packe
 #endif
 }
 
+void MyMesh::onMalformedChannelMessageRecv(const mesh::GroupChannel &channel, mesh::Packet *pkt, uint32_t timestamp,
+                                           const char *reason, uint8_t score) {
+  (void)reason;
+  (void)score;
+  onChannelMessageRecv(channel, pkt, timestamp, TXT_MALFORMED_PLACEHOLDER);
+}
+
 void MyMesh::onChannelDataRecv(const mesh::GroupChannel &channel, mesh::Packet *pkt, uint16_t data_type,
                                const uint8_t *data, size_t data_len) {
   if (data_len > MAX_CHANNEL_DATA_LENGTH) {
@@ -925,7 +940,6 @@ void MyMesh::begin(bool has_display) {
   _prefs.tx_power_dbm = constrain(_prefs.tx_power_dbm, -9, MAX_LORA_TX_POWER);
   _prefs.gps_enabled = constrain(_prefs.gps_enabled, 0, 1);  // Ensure boolean 0 or 1
   _prefs.gps_interval = constrain(_prefs.gps_interval, 0, 86400);  // Max 24 hours
-
 #ifdef BLE_PIN_CODE // 123456 by default
   if (_prefs.ble_pin == 0) {
 #ifdef DISPLAY_CLASS

examples/companion_radio/MyMesh.h

@@ -136,8 +136,12 @@ class MyMesh : public BaseChatMesh, public DataStoreHost {
                          const char *text) override;
   void onSignedMessageRecv(const ContactInfo &from, mesh::Packet *pkt, uint32_t sender_timestamp,
                            const uint8_t *sender_prefix, const char *text) override;
+  void onMalformedMessageRecv(const ContactInfo &from, mesh::Packet *pkt, uint32_t sender_timestamp,
+                              const char *reason, uint8_t score) override;
   void onChannelMessageRecv(const mesh::GroupChannel &channel, mesh::Packet *pkt, uint32_t timestamp,
                             const char *text) override;
+  void onMalformedChannelMessageRecv(const mesh::GroupChannel &channel, mesh::Packet *pkt, uint32_t timestamp,
+                                     const char *reason, uint8_t score) override;
   void onChannelDataRecv(const mesh::GroupChannel &channel, mesh::Packet *pkt, uint16_t data_type,
                          const uint8_t *data, size_t data_len) override;
 

examples/companion_radio/NodePrefs.h

@@ -34,4 +34,4 @@ struct NodePrefs {  // persisted to file
   uint8_t autoadd_max_hops;  // 0 = no limit, 1 = direct (0 hops), N = up to N-1 hops (max 64)
   char default_scope_name[31];
   uint8_t default_scope_key[16];
-};
+};