Skip to content

mavlogdump: keep one-off and burst messages when using --reduce-rate - #1276

Open
azrabano23 wants to merge 1 commit into
ArduPilot:masterfrom
azrabano23:fix-1033-reduce-rate-keep-oneoff-msgs
Open

mavlogdump: keep one-off and burst messages when using --reduce-rate#1276
azrabano23 wants to merge 1 commit into
ArduPilot:masterfrom
azrabano23:fix-1033-reduce-rate-keep-oneoff-msgs

Conversation

@azrabano23

Copy link
Copy Markdown

What was wrong

mavlogdump.py --reduce-rate N drops the parameter download from a tlog, even with --parms (issue #1033). On a synthetic tlog containing 100 ATTITUDE at 50 Hz, a burst of 10 PARAM_VALUE 5 ms apart, 5 STATUSTEXT 5 ms apart, and 1 Hz HEARTBEATs from two components 50 ms apart:

$ mavlogdump.py --reduce-rate 10 in.tlog --output out.tlog --quiet --parms
# message counts in out.tlog, master:
{'ATTITUDE': 17, 'HEARTBEAT': 1}

All 10 PARAM_VALUE and all 5 STATUSTEXT are gone, and one of the two heartbeat sources disappears entirely.

Root cause

reduce_rate_msg() exempts a fixed list of message types from rate reduction, but the list only contains DataFlash names (PARM, MSG, FMT, ...). Their tlog equivalents (PARAM_VALUE, STATUSTEXT) are rate-limited like any streaming message, so a burst is reduced to at most its first element. The --parms handling further down the loop is never reached because the message has already been discarded. The list also has EVT, which is not a DataFlash message; the event message is EV (as the --meta list in the same file already uses).

The fix

Extend the exempt list, following the discussion on the issue:

  • PARAM_VALUE, STATUSTEXT: tlog equivalents of PARM, MSG; sent in bursts, not at a steady rate.
  • MISSION_ITEM_INT, CMD: mission download, same reasoning.
  • HEARTBEAT: sent at 1 Hz by every component; with two sources a few ms apart, rate limiting drops one source completely.
  • EVT -> EV, plus ERR (one-off DataFlash messages, matching MAVExplorer's cmd_messages).

AUTOPILOT_VERSION was left out as agreed on the issue (sent once, so unaffected by rate limiting in practice).

After the change, on the same input:

{'ATTITUDE': 17, 'PARAM_VALUE': 10, 'STATUSTEXT': 5, 'HEARTBEAT': 4}

Verification

Added test_reduce_rate_keeps_oneoff_tlog_messages in tests/test_mavlogdump.py. It writes the synthetic tlog described above, runs mavlogdump.py --reduce-rate 10 --quiet --parms --output, reads the output back with mavutil.mavlink_connection, and asserts all PARAM_VALUE, STATUSTEXT and HEARTBEAT messages survive while ATTITUDE is reduced. It fails on master (assert 0 == 10 for PARAM_VALUE) and passes with this change.

python3 -m pip install .
python3 -m pytest tests/test_mavlogdump.py -q   # 2 passed
flake8 tools/mavlogdump.py tests/test_mavlogdump.py --count --select=E9,F63,F7,F82   # 0

Tested with Python 3.12 on macOS.

Fixes #1033

🤖 Generated with Claude Code

`mavlogdump.py --reduce-rate N` applies the rate limit to every message
type except a short list of DataFlash meta messages. That list only
contained DataFlash names, so on a tlog the parameter download burst
(PARAM_VALUE at a few ms spacing) was reduced to almost nothing, even
with --parms:

    mavlogdump.py --reduce-rate 10 in.tlog --output out.tlog --quiet --parms
    mavlogdump.py --types PARAM_VALUE out.tlog   # no output

Add the tlog equivalents of the exempt DataFlash messages: PARAM_VALUE
(PARM), STATUSTEXT (MSG) and MISSION_ITEM_INT/CMD for missions, and
keep HEARTBEAT so that every source system/component is still present
in the reduced log (two sources beating 1Hz a few ms apart would
otherwise have one of them dropped entirely). Also add ERR and rename
EVT to EV, which is the actual DataFlash event message name (and what
--meta already uses in this file).

Add a regression test that writes a small synthetic tlog with a 50Hz
ATTITUDE stream, PARAM_VALUE and STATUSTEXT bursts and 1Hz heartbeats
from two components, runs mavlogdump.py --reduce-rate 10 --parms on
it, and checks that the bursts and heartbeats survive intact while
ATTITUDE is reduced.

Fixes ArduPilot#1033

Signed-off-by: Azra Bano <azrabano.work@gmail.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

mavlogdump.py discards parameter messages in tlog files unlike in dataflash

1 participant