Add hw_ppe.qos: shape in the switch where the switch offloads the flow - #193
Add hw_ppe.qos: shape in the switch where the switch offloads the flow#193JuliusBairaktaris wants to merge 2 commits into
Conversation
fc46927 to
e39091e
Compare
|
So for what it is worth, I am all for more approaches here and more diverse qos scripts and if this works well for you, you can ignore the rest. |
27ce43c to
6b32d92
Compare
|
@moeller0 saw the comment on the typo, do you want me to fix that? Also please wait until: openwrt/openwrt#24806 is merged. Thanks! |
What typo? |
| if [ ! -z "${TMP}" ] && echo "${TMP}" | grep -q mirred; then | ||
| # oops, a redirect is there but we failed to parse it? Ask for a user report | ||
| sqm_error "#---- CUT HERE ----#" | ||
| sqm_error "get_ifb_associated_with_if failed to extrect the ifb name from:" |
get_ifb_associated_with_if() treats any filter on parent ffff: that does not name an ifb as a parse failure and prints a cut-here block asking the user to file an issue. A script that puts something else on the ingress qdisc - a policer, say - therefore prints that block on every stop. Look for the redirect before deciding the name should have been there. A real parse failure still reports. Signed-off-by: Julius Bairaktaris <julius@bairaktaris.de>
51a4009 to
eaf11c0
Compare
…ower Connections a switch has offloaded to its flow engine never pass a software qdisc again, so a shaper that lives in a qdisc governs an ever-smaller share of the traffic the moment offloading is on. This script shapes in the switch instead, written for and measured on the Qualcomm PPE the qca_ppe driver programs, and driven entirely through existing offload uAPIs: tbf for the egress rate, mqprio for the download band's rate, flower with skbedit for the classification that feeds it, and the driver's cpu_port_rate parameter for the one port no qdisc can name. Egress is a root tbf on the WAN port, which the driver turns into the port's hardware token bucket; the qdisc beneath carries what still travels the CPU path. The tbf limit is the queue depth the driver programs, sized by eshaper_queue_dur_us of the rate - deep enough for a single flow to reach the shaped rate, while frames the switch classifies as latency-sensitive ride higher-priority queues past it. The download is shaped where it leaves rather than where it arrives - the driver reaches egress queues and an ingress meter, and a meter drops where a queue delays - and only it. Every user port's scheduler keeps a band of queues no frame selects on its own; a flower rule on the WAN port marks each arriving frame addressed to the router - every routed download frame, nothing bridged - and the mark selects that band on whichever port the frame leaves by. An mqprio class spanning the band carries the download rate, which the driver programs into the band's scheduler node. A transfer between two local ports never wears the mark and passes at line rate. What the switch does not forward in hardware leaves for the host by the CPU port, and only that: the flows the flow table does not hold, and the path to Wi-Fi clients. That port has no netdev, so the download rate reaches its band through the driver's cpu_port_rate parameter and the switch shapes that remainder too; no packet is redirected to an ifb and nothing is shaped by the host. Local traffic to a Wi-Fi client crosses the CPU port unmarked and is not held. The user ports' bands and the CPU port's each carry the full download rate, so a workload split across both can exceed the rate while the split lasts; sustaining that needs a flow table too full to take new flows. The mark sits at preference 511, the lowest standing the switch holds, so that every classified rule ppe-qos installs overrides it and a DNS answer or a game packet keeps its own queue instead of the band's. Measured on an IPQ8074 AX3600 (kernel 6.18, PPPoE line shaped 305/165 Mbit/s), against the same script shaping every user port's egress instead, arms interleaved run for run. A wired transfer between two switch ports reads 941/940/939 Mbit/s where a per-port egress shaper holds it to 287/287/287, against an unshaped 932-939 baseline. The download direction is unchanged: 8-stream download 286.1/286.0/286.1 against 286.1/286.1/286.0 Mbit/s, ICMP under that load 19.54/19.62/19.59 against 19.69/19.55/19.63 ms, idle 19.90 ms. Run concurrently, the download holds 285.0 Mbit/s at 19.7 ms while the local transfer runs 915-934 Mbit/s beside it. The switch's per-queue counters attribute the download to the band's four queues (1506-byte average) with the acks promoted past it (80-byte average). The path to a Wi-Fi client, measured on the same board with an HE160 client downloading from the uplink, three interleaved runs per arm: shaped by the CPU port's band at 305000 the client reads 239.6/235.9/241.7 Mbit/s with the router's four cores 11.6/11.4/11.9% busy, where the ifb this replaces read 278.5/278.7/276.3 Mbit/s at 35.6/35.8/36.4%; unshaped, 311.6/318.5/319.6 Mbit/s at 14.4/16.9/14.1%. At 500000, above the line, the client reads 311.9/318.0/321.0 Mbit/s - the rate that would show a shaper that binds where it should not. The download band and the mqprio rate need a driver that serves internal priorities eight and up from a schedulable node of their own, and a cpu_port_rate that shapes that band on the CPU port; the qca_ppe driver does. On a driver without the band the mqprio install fails and the script fails loudly with it, the same stance skip_sw takes for the filter. Assisted-by: Claude:claude-fable-5 Signed-off-by: Julius Bairaktaris <julius@bairaktaris.de>
eaf11c0 to
7c6ba27
Compare
A connection the kernel has offloaded to a switch's flow engine is never handed to a qdisc again, so a shaper built on cake and an ifb does not shape it — the CPU never sees those packets. Where the switch can be told a rate instead, it can still be governed.
hw_ppe.qosdoes that: a roottbfon the port, which a driver implementingTC_SETUP_QDISC_TBFprograms into the port's hardware token bucket, and amatchallpolicer on the port's ingress in place of the redirect to an ifb — a switch cannot queue an arriving packet in order to shape it, but it can meter one. No ifb is created and nothing is redirected. The ingress filter isskip_sw, so a kernel or switch that cannot take it fails loudly instead of quietly policing on the CPU.Two defaults do the heavy lifting, both measured:
limitof 1 ms of the shaped rate (never less than eight full frames) instead of the customarylatency 300ms. The limit is how deep the queue behind the shaper may get, and a driver that drains a real queue sizes that queue from it: at 165 Mbit/s the wide limit held 5.4 ms of standing queue under upload saturation — the bufferbloat the shaper was installed against — where the 1 ms limit holds the added latency under measurement noise at the same throughput.The ingress side is still a trade (thanks @moeller0 for keeping this honest): a policer drops where a shaper would have delayed, so it adds no queueing delay but not no delay, and ECN cannot be honoured. What it buys is that offloaded connections are governed at all, at zero CPU. Raise
ishaper_burst_dur_usto tolerate more, or setdownloadto 0 to keep only hardware egress shaping.Configure it on the physical port (
wan), not on a pppoe or vlan device above it — the hardware only knows the port. That placement is also what covers traffic the switch cannot offload, Wi-Fi included: it shares the port.What a user can configure
All ordinary
/etc/config/sqmoptions;run-openwrt.shnow exports the burst durations, so those work from uci (and LuCI's dropdown picks the script up from/usr/lib/sqm/).interfacepppoe-wanor a VLAN above itupload/downloadeshaper_burst_dur_usishaper_burst_dur_usqdisc,elimit,etarget,eecn,eqdisc_optscakefalls back tofq_codelwith a warninglinklayer,overhead,tcMTU,tcMPU,tcTSIZEilimit,itarget,iecn,iqdisc_optsTesting
Qualcomm IPQ8074 (Xiaomi AX3600), kernel 6.18, a driver offloading both directions, on a 336 Mbit/s PPPoE line shaped to 335000/165000. Latency probed at 20 Hz from a wired LAN host, load from public speed endpoints:
Zero added latency at 96–97% of the configured rates, both directions loaded. The rates and the queue limit read back exactly in the hardware registers;
stopclears the token bucket and the meter (verified by register readback); the uci burst knobs were verified end to end (option → filter burst → register).The functions.sh change
get_ifb_associated_with_if()treats any filter onparent ffff:that does not name an ifb as a parse failure and prints the cut-here block asking the user to open an issue. It now looks for the redirect before deciding the name should have been there; a real parse failure still reports.