feat(logging): add log console view (LogCapture ring buffer, REST /api/logs, MQTT event export) - #169
Conversation
Replace direct Serial.println/printf calls across all 21 source files with the LOG_DEBUG/LOG_INFO/LOG_WARN/LOG_ERROR macros so output flows through the ring buffer and becomes visible in the web log console. LogCapture.cpp keeps its internal Serial mirror as documented fallback.
RelayModuleNode and Rule* use the LOG_* macros since the Serial->LogCapture migration; without LogCapture.cpp the target fails to link.
Unauthenticated GET /api/logs returns captured ring-buffer entries as JSON (since/count/level query args, next-sequence for polling); authenticated POST /api/logs/clear empties the buffer. buildLogsJson is public for native testing.
The real ESP32 UpdateClass::errorString() returns const char*, but the native mock returned String. Passing a String through the variadic LOG_ERROR macro triggered -Wnon-pod-varargs in the host build; matching the real API keeps both the native and PlatformIO builds clean.
…A discovery Task 5 of the logging-view plan: publish a Home Assistant 'event' entity discovery payload (platform=event, curated event_types whitelist) and an export pump inside publishStates() that publishes new LogCapture entries since the last exported sequence as HA event messages. WARN/ERROR entries are additionally mirrored to the raw pool-controller/log topic with seq/t/level/msg fields. Boot backlog is skipped by initializing the last exported sequence in begin().
Native Test Coverage
Report from native unit tests (ASan + gcov). |
✅MegaLinter analysis: Success
See detailed reports in MegaLinter artifacts Your project could benefit from a custom flavor, which would allow you to run only the linters you need, and thus improve runtime performances. (Skip this info by defining
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 042643036d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Native Test Coverage
Report from native unit tests (ASan + gcov). |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cf949d3d4c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Boot-time LOG_* entries (WiFi/MQTT init, sensor scans) were cleared by LogCapture::begin() running after context.setup(), and the pre-reset sequence was saved as MQTT export watermark causing skipped events. Moves begin() right after the serial wait, before context.setup(). Addresses PR #169 review Ci2.
getEntries() treats since as exclusive (skips entry.seq <= since), so returning lastSeq()+1 made the client skip the entry whose seq equals that cursor, and truncated responses jumped past unreturned entries. next now identifies the highest consumed seq (or stays at since when nothing was consumed). Addresses PR #169 review Ci3.
LogCapture::lastSeq() could be newer than the captured snapshot when an async WiFi/MQTT callback appended a log mid-export, marking the concurrent entry as exported without ever processing it (permanent HA event loss). Watermark now uses the last seq of the completed snapshot. Addresses PR #169 review Ci4.
The unconditional 2s loadLogs timer kept fetching and appending nodes in hidden tabs, growing the log DOM by tens of thousands of nodes per day. Polling now runs only while the document is visible and the Logs tab is active; oldest entries are evicted beyond a fixed 500-node client cap. Addresses PR #169 review K4g.
Native Test Coverage
Report from native unit tests (ASan + gcov). |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2ce3c2206d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… dedupe web polls - LogCapture: format into a 512-byte LOG_FORMAT_SIZE buffer so the Serial mirror receives the complete message while the ring copy is truncated to LOG_MSG_SIZE-1 in store() (VprqH) - MqttPublisher: advance the export watermark only through entries whose required publishes were successfully queued; a failed publish keeps the watermark before the entry so the tail is retried on the next pass instead of being dropped (VprqL) - app.js: generation token on log polls — stale in-flight responses are discarded after a newer poll, filter switch, or clear, preventing duplicate/stale entries and lastLogSeq overwrites (VprqO) - tests: Serial capture mock + full-mirror/ring-truncation test; publish-failure watermark retry test
Native Test Coverage
Report from native unit tests (ASan + gcov). |
- Define SerialClass::s_capture / s_captureEnabled in relay_safety stubs.cpp so the binary links after LogCapture's Serial mirror references the mock statics (captures.cpp is not part of that target). - clang-format fixes in test_logcapture.cpp / test_mqttpublisher.cpp.
Native Test Coverage
Report from native unit tests (ASan + gcov). |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4e32104397
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
getEntries() treats a sinceSeq higher than lastSeq() (a client cursor persisted across a reboot, where begin() restarts the sequence at 0) as a cursor reset and returns the whole currently-visible ring. Without this, boot diagnostics stay invisible until the new sequence wraps past the stale cursor, because the empty response preserves the cursor.
Migrate the remaining direct Serial writes in SystemMonitor (critical/low memory warnings, reboot request, boot-loop detection) to LOG_* macros so they appear in /api/logs and the MQTT warning export instead of only on the serial console.
Native Test Coverage
Report from native unit tests (ASan + gcov). |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 85154c3d07
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Native Test Coverage
Report from native unit tests (ASan + gcov). |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1b8d730d7b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Native Test Coverage
Report from native unit tests (ASan + gcov). |
A polling client's cursor (sinceSeq) is only trustworthy within the boot that produced it. The previous seq-only clamp (sinceSeq > s_seq) missed reboots where the new boot already produced more entries than the stale cursor (e.g. cursor 20, new boot at seq 40) and silently skipped the new boot's entries 1..20. begin() now increments a monotonic boot epoch; getEntries() treats any cursor with a mismatching epoch (or a seq past lastSeq()) as stale and returns the whole ring. /api/logs echoes the current epoch as 'boot' so clients can detect the reboot even while the new sequence is below their stored cursor. Tests: 3 new LogCapture::reboot_epoch suites (epoch changes across reboots; stale pre-reboot cursor returns the full ring; current-boot cursor still filters).
A poll whose fetch() response takes longer than the 2s tick was superseded by the next tick's request: the new request bumped the generation token, so the slow response was discarded and the console stopped updating until a clear or filter change. A logPollInFlight flag now serializes polls — the next tick resumes once the in-flight one settles. Also send the cursor's boot epoch as 'boot' and adopt the server's 'boot' value, so after a reboot (where the server re-sends the whole ring) the client drops the stale pre-reboot lines instead of appending on top of them.
Native Test Coverage
Report from native unit tests (ASan + gcov). |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: db057dbcd9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Review feedback P2: - A static RAM epoch restarts at the same value after a real reboot (0 then +1), so it cannot distinguish physical boots. On ESP32 begin() now assigns a fresh random epoch via esp_random(); native test builds keep a deterministic monotonic increment. - Clients that omit the boot query parameter previously got epoch 0 while the logger reports epoch 1, breaking the documented since-polling flow (full dump on every request). Omitted boot now means current boot, restoring incremental polling; explicit stale epochs still force a full dump.
Native Test Coverage
Report from native unit tests (ASan + gcov). |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: edfe4e2780
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Addresses PR #169 review comments. - WebPortal::apiGetLogs: parse the boot query parameter with strtoul and validate the uint32 range. esp_random() produces values above LONG_MAX on roughly half of boots, which atol() overflowed into a stale epoch, forcing a full ring + duplicate entries on every poll. Invalid input keeps the current boot (safe fallback preserving incremental since-polling). - MqttPublisher::begin: start the export watermark at seq 0 instead of lastSeq() so WARN/ERROR and curated events emitted before the publisher starts (boot-loop detection, ConfigManager, WPS/SSID warnings) are not permanently dropped. exportLogEvents() already filters Info/Debug chatter. - Add test: pre-publisher WARN is exported via MQTT.
Native Test Coverage
Report from native unit tests (ASan + gcov). |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 70bb1aec6c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| case LogLevel::Error: | ||
| eventType = "LOG_ERROR"; | ||
| break; | ||
| default: |
There was a problem hiding this comment.
When a module records the supported LogLevel::Critical, this switch takes the default branch, advances lastOkSeq, and permanently omits the entry from both the Home Assistant event entity and the raw log topic. Since critical is a severity above warning and is exposed by the logging API, handle it as LOG_ERROR or provide a dedicated critical event type rather than treating it like Info chatter.
Useful? React with 👍 / 👎.
1 participant
What
Adds a complete logging view to the pool controller:
LogCapture(level-aware)GET /api/logswith filters (since,count,level)logEventmarkers (mode/pump/wifi/mqtt transitions) published as an MQTT event entity with Home Assistant discovery, plus a sample HA Logbook automation in the docsWhy
Previously all log output went to the serial line only; there was no way to inspect runtime events on-device, remotely via REST, or in Home Assistant. This lays the groundwork for the planned dashboard logging features and makes 24/7 operation observable.
Test plan
LogCapture(pio test -e native)Commits
13 commits from the prepared logging work (spec docs → core → boot init → serial migration → REST → MQTT → web console), followed by
feat(web): add log console view with level filter and clear.