mavlogdump: keep one-off and burst messages when using --reduce-rate - #1276
Open
azrabano23 wants to merge 1 commit into
Open
mavlogdump: keep one-off and burst messages when using --reduce-rate#1276azrabano23 wants to merge 1 commit into
azrabano23 wants to merge 1 commit into
Conversation
`mavlogdump.py --reduce-rate N` applies the rate limit to every message
type except a short list of DataFlash meta messages. That list only
contained DataFlash names, so on a tlog the parameter download burst
(PARAM_VALUE at a few ms spacing) was reduced to almost nothing, even
with --parms:
mavlogdump.py --reduce-rate 10 in.tlog --output out.tlog --quiet --parms
mavlogdump.py --types PARAM_VALUE out.tlog # no output
Add the tlog equivalents of the exempt DataFlash messages: PARAM_VALUE
(PARM), STATUSTEXT (MSG) and MISSION_ITEM_INT/CMD for missions, and
keep HEARTBEAT so that every source system/component is still present
in the reduced log (two sources beating 1Hz a few ms apart would
otherwise have one of them dropped entirely). Also add ERR and rename
EVT to EV, which is the actual DataFlash event message name (and what
--meta already uses in this file).
Add a regression test that writes a small synthetic tlog with a 50Hz
ATTITUDE stream, PARAM_VALUE and STATUSTEXT bursts and 1Hz heartbeats
from two components, runs mavlogdump.py --reduce-rate 10 --parms on
it, and checks that the bursts and heartbeats survive intact while
ATTITUDE is reduced.
Fixes ArduPilot#1033
Signed-off-by: Azra Bano <azrabano.work@gmail.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was wrong
mavlogdump.py --reduce-rate Ndrops the parameter download from a tlog, even with--parms(issue #1033). On a synthetic tlog containing 100 ATTITUDE at 50 Hz, a burst of 10 PARAM_VALUE 5 ms apart, 5 STATUSTEXT 5 ms apart, and 1 Hz HEARTBEATs from two components 50 ms apart:All 10 PARAM_VALUE and all 5 STATUSTEXT are gone, and one of the two heartbeat sources disappears entirely.
Root cause
reduce_rate_msg()exempts a fixed list of message types from rate reduction, but the list only contains DataFlash names (PARM,MSG,FMT, ...). Their tlog equivalents (PARAM_VALUE,STATUSTEXT) are rate-limited like any streaming message, so a burst is reduced to at most its first element. The--parmshandling further down the loop is never reached because the message has already been discarded. The list also hasEVT, which is not a DataFlash message; the event message isEV(as the--metalist in the same file already uses).The fix
Extend the exempt list, following the discussion on the issue:
PARAM_VALUE,STATUSTEXT: tlog equivalents ofPARM,MSG; sent in bursts, not at a steady rate.MISSION_ITEM_INT,CMD: mission download, same reasoning.HEARTBEAT: sent at 1 Hz by every component; with two sources a few ms apart, rate limiting drops one source completely.EVT->EV, plusERR(one-off DataFlash messages, matching MAVExplorer'scmd_messages).AUTOPILOT_VERSIONwas left out as agreed on the issue (sent once, so unaffected by rate limiting in practice).After the change, on the same input:
Verification
Added
test_reduce_rate_keeps_oneoff_tlog_messagesintests/test_mavlogdump.py. It writes the synthetic tlog described above, runsmavlogdump.py --reduce-rate 10 --quiet --parms --output, reads the output back withmavutil.mavlink_connection, and asserts all PARAM_VALUE, STATUSTEXT and HEARTBEAT messages survive while ATTITUDE is reduced. It fails on master (assert 0 == 10for PARAM_VALUE) and passes with this change.Tested with Python 3.12 on macOS.
Fixes #1033
🤖 Generated with Claude Code