Issue during scrape qdiscs RED

[Mickaelh51]

Hi,
first, thanks a lot for your work, it's very interesting !

I have an issue when I want to scrape qdiscs

time=2025-04-22T12:51:08.473+02:00 level=DEBUG msg="starting metrics scrape"
time=2025-04-22T12:51:08.480+02:00 level=ERROR msg="failed to get qdiscs" interface=bond0 err="unmarshalRed()\t4\n\t[0 0 0 0 15 0 0 0]"
time=2025-04-22T12:51:08.491+02:00 level=ERROR msg="failed to get qdiscs" interface=bond0 err=EOF
time=2025-04-22T12:51:08.491+02:00 level=DEBUG msg="metric scrape complete"

I use: tc_exporter_0.8.3_linux_amd64

you can see my qdiscs:

qdisc htb 1: root refcnt 17 r2q 12500 default 0x33 direct_packets_stat 0 direct_qlen 1000
 Sent 4854761 bytes 84238 pkt (dropped 0, overlimits 84239 requeues 0) 
 backlog 0b 0p requeues 0
qdisc sfq 801c: parent 1:28 limit 127p quantum 9014b depth 127 divisor 1024 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc sfq 801a: parent 1:14 limit 127p quantum 9014b depth 127 divisor 1024 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc red 8020: parent 1:33 limit 72Kb min 9Kb max 18Kb 
 Sent 4854703 bytes 84237 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  marked 0 early 0 pdrop 0 other 0 
qdisc sfq 801b: parent 1:1e limit 127p quantum 9014b depth 127 divisor 1024 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc sfq 801d: parent 1:32 limit 127p quantum 9014b depth 127 divisor 1024 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 801f: parent 1:a limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb drop_batch 64 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0

issue comes from unmarshalRed from q_red.go in go-tc project

can you help me to find the cause of this issue please ?

thanks in advance

[fbegyn]

Hmmm, I’ll see if I can’t find some spare time this week(end) to look into this.

Could you provide some more info about the OS you are running (flavour and kernel versions) and perhaps some example (script) on how you set up the qdiscs? That way I can throw it against a dev VM.

If you haven’t, perhaps try a version lower than v0.8.x. It would be interesting to see if there was a regression somewhere (the test suite isn’t extensive and I did switch some things around).

[Mickaelh51]

Thanks for your answer
I tested with 0.8.0 it's the same issue.

it's vyos, it's a Debian 12 base and kernel: 6.6.66-vyos

Version:          VyOS 1.5-stream-2025-Q1
Release train:    circinus
Release flavor:   generic

Built by:         VyOS Networks Iberia S.L.U.
Built on:         Thu 13 Feb 2025 18:06 UTC
Build UUID:       b38b28e0-a516-4f56-a596-5502ae094d3b
Build commit ID:  5128f5e45cdb73-dirty

Architecture:     x86_64
Boot via:         installed image
System type:      KVM guest

Hardware vendor:  QEMU
Hardware model:   Ubuntu 24.04 PC (i440FX + PIIX, 1996)

Copyright:        VyOS maintainers and contributors

on vyos I added these rules:

set interface bond0 egress 'EGRESSQOS'
set policy shaper EGRESSQOS bandwidth '20gbit'
set policy shaper EGRESSQOS class 2 bandwidth '20%'
set policy shaper EGRESSQOS class 2 ceiling '100%'
set policy shaper EGRESSQOS class 2 description 'qACK'
set policy shaper EGRESSQOS class 2 match-group 'qACK'
set policy shaper EGRESSQOS class 2 priority '0'
set policy shaper EGRESSQOS class 2 queue-type 'fair-queue'
set policy shaper EGRESSQOS class 10 bandwidth '10%'
set policy shaper EGRESSQOS class 10 ceiling '100%'
set policy shaper EGRESSQOS class 10 description 'qRTP'
set policy shaper EGRESSQOS class 10 match-group 'qRTP'
set policy shaper EGRESSQOS class 10 priority '1'
set policy shaper EGRESSQOS class 10 queue-type 'fq-codel'
set policy shaper EGRESSQOS default bandwidth '43%'
set policy shaper EGRESSQOS default ceiling '100%'
set policy shaper EGRESSQOS default priority '7'
set policy shaper EGRESSQOS default queue-type 'random-detect'
set traffic-match-group qACK match ACK ip tcp ack
set traffic-match-group qACK match SYNACK ip tcp syn
set traffic-match-group qRTP match RTP-DSCP ip dscp 'EF'
set traffic-match-group qRTP match RTP-DSCP ip protocol 'udp'

and vyos transforms these rules by classic TC commands:

vyos@am-rt-001# tc qdisc show dev bond0
qdisc htb 1: root refcnt 17 r2q 12500 default 0xb direct_packets_stat 0 direct_qlen 1000
qdisc sfq 8001: parent 1:2 limit 127p quantum 9014b depth 127 divisor 1024 
qdisc red 8005: parent 1:b limit 72Kb min 9Kb max 18Kb 
qdisc fq_codel 8004: parent 1:a limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb drop_batch 64 

vyos@am-rt-001# tc class show dev bond0
class htb 1:1 root rate 20Gbit ceil 20Gbit burst 0b cburst 0b
class htb 1:2 parent 1:1 leaf 8001: prio 0 rate 4Gbit ceil 1Gbit burst 14500b cburst 1375b
class htb 1:a parent 1:1 leaf 8004: prio 1 rate 2Gbit ceil 1Gbit burst 15000b cburst 1375b
class htb 1:b parent 1:1 leaf 8005: prio 7 rate 8600Mbit ceil 20Gbit burst 13975b cburst 0b
class red 8005:1 parent 8005: 

vyos@am-rt-001# tc filter show dev bond0
filter parent 1: protocol all pref 1 u32 chain 0 
filter parent 1: protocol all pref 1 u32 chain 0 fh 800: ht divisor 1 
filter parent 1: protocol all pref 1 u32 chain 0 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1:a not_in_hw 
  match 00110000/00ff0000 at 8
  match 00b80000/00ff0000 at 0
filter parent 1: protocol all pref 1 u32 chain 0 fh 800::801 order 2049 key ht 800 bkt 0 flowid 1:a not_in_hw 
  match 00110000/00ff0000 at 8
  match 00b80000/00ff0000 at 0
	action order 1:  police 0x1 rate 100Mbit burst 15337b mtu 2Kb action reclassify overhead 0b 
	ref 1 bind 1 

I think it's not necessary to reproduce with Vyos, just a debian 12 and these TC commands.

thanks in advance.
I have to do the same thing (reproduce with debian 12 only)

UPDATE:
I tested on simple debian 12.6:
6.1.0-22-cloud-amd64
same result

add qdisk manually:

root@debian:/home/debian# tc qdisc show dev ens4
qdisc htb 1: root refcnt 2 r2q 12500 default 0x11 direct_packets_stat 3798 direct_qlen 1000
qdisc sfq 8001: parent 1:2 limit 127p quantum 1514b depth 127 divisor 1024 
qdisc red 8005: parent 1:b limit 72Kb min 9Kb max 18Kb 
qdisc fq_codel 8004: parent 1:a limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64 

same issue with 0.8.3:

root@debian:/home/debian# ./tc_exporter_0.8.3_linux_amd64 
{map[default:{[ens4]}] {[] []}}
time=2025-04-25T10:22:24.589Z level=INFO msg="making qdisc collector" collector=qdisc
time=2025-04-25T10:22:24.592Z level=INFO msg="making class collector" collector=class
time=2025-04-25T10:22:24.592Z level=INFO msg="starting TC exporter" listen-address=:9704
time=2025-04-25T10:23:23.059Z level=ERROR msg="failed to get qdiscs" interface=ens4 err="unmarshalRed()\t4\n\t[0 0 0 0 15 0 0 0]"
time=2025-04-25T10:23:23.078Z level=ERROR msg="failed to get qdiscs" interface=ens4 err=EOF

[weidongkl]

The error seems to originate from q_red.go in the github.com/florianl/go-tc library.

[weidongkl]

Would you be able to verify if this pull request resolves your issue? @Mickaelh51

[fbegyn]

@Mickaelh51 when you have the chance, could you make a build from main to check if that fixed the RED qdiscs? Some quick testing on VM and on my own router seems to deliver functional metrics without the exporter crashing.

May 12 23:59:37 router-01 tc-exporter[196199]: time=2025-05-12T23:59:37.593+02:00 level=DEBUG msg="qdisc type" kind=red handle=2151088128
May 12 23:59:37 router-01 tc-exporter[196199]: time=2025-05-12T23:59:37.593+02:00 level=DEBUG msg="passing qdisc to qdisc collector" qdisc="{Msg:{Family:0 Ifindex:7 Handle:2151088128 Parent:4294967295 Info:2} Attribute:{Kind:red EgressBlock:<nil> IngressBlock:<nil> HwOffload:0xc00033572c Chain:<nil> Stats:0xc00002b9b0 XStats:0xc000344420 Stats2:0xc000126840 Stab:<nil> ExtWarnMsg: Basic:<nil> BPF:<nil> Cgroup:<nil> U32:<nil> Rsvp:<nil> Route4:<nil> Fw:<nil> Flow:<nil> Flower:<nil> Matchall:<nil> TcIndex:<nil> Cake:<nil> FqCodel:<nil> Codel:<nil> Fq:<nil> Pie:<nil> Hhf:<nil> Tbf:<nil> Sfb:<nil> Sfq:<nil> Red:0xc00034a210 MqPrio:<nil> Pfifo:<nil> Bfifo:<nil> Choke:<nil> Netem:<nil> Plug:<nil> Cbs:<nil> Htb:<nil> Hfsc:<nil> HfscQOpt:<nil> Dsmark:<nil> Drr:<nil> Cbq:<nil> Atm:<nil> Qfq:<nil> Prio:<nil> TaPrio:<nil>}}"
May 12 23:59:37 router-01 tc-exporter[196199]: time=2025-05-12T23:59:37.593+02:00 level=DEBUG msg="passing qdisc to red collector" qdisc="{Msg:{Family:0 Ifindex:7 Handle:2151088128 Parent:4294967295 Info:2} Attribute:{Kind:red EgressBlock:<nil> IngressBlock:<nil> HwOffload:0xc00033572c Chain:<nil> Stats:0xc00002b9b0 XStats:0xc000344420 Stats2:0xc000126840 Stab:<nil> ExtWarnMsg: Basic:<nil> BPF:<nil> Cgroup:<nil> U32:<nil> Rsvp:<nil> Route4:<nil> Fw:<nil> Flow:<nil> Flower:<nil> Matchall:<nil> TcIndex:<nil> Cake:<nil> FqCodel:<nil> Codel:<nil> Fq:<nil> Pie:<nil> Hhf:<nil> Tbf:<nil> Sfb:<nil> Sfq:<nil> Red:0xc00034a210 MqPrio:<nil> Pfifo:<nil> Bfifo:<nil> Choke:<nil> Netem:<nil> Plug:<nil> Cbs:<nil> Htb:<nil> Hfsc:<nil> HfscQOpt:<nil> Dsmark:<nil> Drr:<nil> Cbq:<nil> Atm:<nil> Qfq:<nil> Prio:<nil> TaPrio:<nil>}}"

[Mickaelh51]

Yep I will try to do this today.
Thanks

[Mickaelh51]

Hi,
I tested, it's OK I can catch some data without last issue, but I don't have all data.

I started my build with:

./tc_exporter --collector-qdisc --collector-class --collector-cbq --collector-codel --collector-fq --collector-fqcodel --collector-hfsc --collector-htb --collector-red --collector-sfb --collector-sfq --log-level="debug"

conf:

listen-address = ":9704"
log-level = 0

[netns.default]
interfaces = ['eth0']

logs:

time=2025-05-13T15:09:23.220+02:00 level=DEBUG msg="starting metrics scrape"
time=2025-05-13T15:09:23.220+02:00 level=DEBUG msg="qdisc type" kind=htb handle=65536
time=2025-05-13T15:09:23.220+02:00 level=DEBUG msg="XStats struct is empty for this qdisc" qdisc="{Msg:{Family:0 Ifindex:2 Handle:65536 Parent:4294967295 Info:3} Attribute:{Kind:htb EgressBlock:<nil> IngressBlock:<nil> HwOffload:0xc00021a608 Chain:<nil> Stats:0xc000031530 XStats:<nil> Stats2:0xc00002bb80 Stab:<nil> ExtWarnMsg: Basic:<nil> BPF:<nil> Cgroup:<nil> U32:<nil> Rsvp:<nil> Route4:<nil> Fw:<nil> Flow:<nil> Flower:<nil> Matchall:<nil> TcIndex:<nil> Cake:<nil> FqCodel:<nil> Codel:<nil> Fq:<nil> Pie:<nil> Hhf:<nil> Tbf:<nil> Sfb:<nil> Sfq:<nil> Red:<nil> MqPrio:<nil> Pfifo:<nil> Bfifo:<nil> Choke:<nil> Netem:<nil> Plug:<nil> Cbs:<nil> Htb:0xc00026c180 Hfsc:<nil> HfscQOpt:<nil> Dsmark:<nil> Drr:<nil> Cbq:<nil> Atm:<nil> Qfq:<nil> Prio:<nil> TaPrio:<nil>}}" interface=eth0

....

time=2025-05-13T15:09:23.221+02:00 level=DEBUG msg="passing qdisc to fq_codel collector" qdisc="{Msg:{Family:0 Ifindex:2 Handle:2148270080 Parent:65546 Info:1} Attribute:{Kind:fq_codel EgressBlock:<nil> IngressBlock:<nil> HwOffload:0xc00021a809 Chain:<nil> Stats:0xc000031d10 XStats:0xc00021f9e0 Stats2:0xc00002bd00 Stab:<nil> ExtWarnMsg: Basic:<nil> BPF:<nil> Cgroup:<nil> U32:<nil> Rsvp:<nil> Route4:<nil> Fw:<nil> Flow:<nil> Flower:<nil> Matchall:<nil> TcIndex:<nil> Cake:<nil> FqCodel:0xc00021f980 Codel:<nil> Fq:<nil> Pie:<nil> Hhf:<nil> Tbf:<nil> Sfb:<nil> Sfq:<nil> Red:<nil> MqPrio:<nil> Pfifo:<nil> Bfifo:<nil> Choke:<nil> Netem:<nil> Plug:<nil> Cbs:<nil> Htb:<nil> Hfsc:<nil> HfscQOpt:<nil> Dsmark:<nil> Drr:<nil> Cbq:<nil> Atm:<nil> Qfq:<nil> Prio:<nil> TaPrio:<nil>}}"
time=2025-05-13T15:09:23.221+02:00 level=ERROR msg="failed to get qdiscs" interface=eth0 err=EOF
time=2025-05-13T15:09:23.221+02:00 level=DEBUG msg="metric scrape complete"

I can parse all my qdisc, but there are errors:

• level=ERROR msg="failed to get qdiscs" interface=eth0 err=EOF
• XStats struct is empty for this qdisc

and in url /metrics I can see all data from:

• qdisc (tc_qdisc_backlog_total, tc_qdisc_packets_total counter, etc ...)
• RED (tc_red_early counter, tc_red_marked RED marked, etc ...)
• fq_codel (tc_fq_codel_ce_mark, tc_fq_codel_drop_overlimit, etc ...)

but no info about class or others options.
Can you help me please ?
Should I have open another issue ?

UPDATE: same issue only with ./tc_exporter --log-level="debug"

thanks in advance

[weidongkl]

It is a normal phenomenon for the xstats of qdisc to be empty. As for the class, I believe it needs to be configured to appear. The error message in the log might be better changed to info.

[weidongkl]

You can use the tc query command to compare. If tc shows it exists but the exporter fails to display it, we need to investigate the reason in detail.

[Mickaelh51]

Thanks for your answer,
I think it's normal, yes, because there is no xstats for all qdisc

$ tc -s -d qdisc show dev eth0
qdisc htb 1: root refcnt 3 r2q 12500 default 0xb direct_packets_stat 0 ver 3.17 direct_qlen 1000
 Sent 787237157 bytes 10186894 pkt (dropped 0, overlimits 10185857 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 8016: parent 1:a limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb drop_batch 64 
 Sent 82620 bytes 918 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 90 drop_overlimit 0 new_flow_count 917 ecn_mark 0
  new_flows_len 1 old_flows_len 0
qdisc sfq 8013: parent 1:2 limit 127p quantum 1514b depth 127 flows 128 divisor 1024 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc red 8017: parent 1:b limit 72Kb min 9Kb max 18Kb ewma 3 probability 0.1 Scell_log 13
 Sent 787154537 bytes 10185976 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  marked 0 early 0 pdrop 0 other 0 

[fbegyn]

Finally reserving some time to take a look at this.

I spotted the eth0 .. err=EOF in my own deployment as well, so that might be worth a seperate issue. Looking into that this weekend.

About the XStat is empty for this qdisc, as you pointed out that makes sense (not every objects has xstats).