set: fix byte order metadata for host-endian types and map data

[anthonyrisinger]

Fix two issues with set/map userdata byte order metadata that cause nft list ruleset to misinterpret element keys and data values on little-endian systems. The kernel-level data is always correct — these are display/round-trip issues affecting the userdata TLVs that the nft CLI uses to reconstruct human-readable output.

Bug 1: Anonymous sets ignore explicit KeyByteOrder

Anonymous/constant/interval sets unconditionally emit KEYBYTEORDER=2 (big-endian), ignoring the KeyByteOrder field. The nft C tool always uses the key expression's actual byte order (mnl.c:mnl_nft_set_add), not a blanket default.

For host-endian types like mark_type (BYTEORDER_HOST_ENDIAN) and ifname_type (BYTEORDER_HOST_ENDIAN), this causes nft list ruleset to misinterpret the element data on LE systems:

• Mark sets: meta mark { 0x00000000, 0x01000000 } instead of { 0x00000000, 0x00000001 }
• Interface name vmaps: iifname vmap { "" : jump chain1 } instead of { "eth0" : jump chain1 }

The existing comment "Semantically useless - kept for binary compatability with nft" is incorrect — KEYBYTEORDER IS semantically meaningful for host-endian types. The nft C tool uses it in netlink_delinearize_setelem to decide whether to call mpz_switch_byteorder on element keys.

Fix: When the anonymous/constant/interval condition fires, check if KeyByteOrder is explicitly set to NativeEndian and emit 1 (host) instead of 2 (big). Falls back to the original big-endian default when KeyByteOrder is unset (zero value), preserving backwards compatibility.

Bug 2: Maps never emit DATABYTEORDER

The NFTNL_UDATA_SET_DATABYTEORDER constant exists in userdata.go but is never used. The Set struct has no DataByteOrder field, and the userdata construction has no code path that emits this TLV.

The nft C tool emits DATABYTEORDER for all datamaps:

if (set_is_datamap(set->flags))
    nftnl_udata_put_u32(udbuf, NFTNL_UDATA_SET_DATABYTEORDER,
                         set->data->byteorder);

Without it, nft list ruleset uses BYTEORDER_INVALID for map data, skips the mpz_switch_byteorder call, and displays native-endian values as byte-swapped on LE systems:

# Expected:
elements = { ... : 0x00000001 }

# Without DATABYTEORDER on LE:
elements = { ... : 0x01000000 }

Fix: Add DataByteOrder field to Set struct and emit NFTNL_UDATA_SET_DATABYTEORDER in the userdata construction for maps when set.

Discovery context

Found while debugging a WireGuard mesh overlay that uses meta mark sets and maps for zone-based connection latching. The incorrect display made it extremely difficult to diagnose whether mark values were being set correctly, since nft list ruleset showed byte-swapped values but the kernel data was actually correct (verified via nft --debug=netlink list ruleset).

Testing

Verified on linux/arm64 (little-endian) that after this patch:

• Anonymous TypeMark sets display correct values (0x00000001 not 0x01000000)
• Map data with TypeMark displays correct values
• nft --debug=netlink list ruleset shows identical kernel-level bytes before and after (no functional change)
• GOOS=linux go build . and GOOS=linux go vet . pass cleanly

[anthonyrisinger]

Company decided they wanted the PR from here—apologies for the noise from the last two closed!—I couldn't figure out how to open #353 again; it was the same exact commit on all three.

[nickgarlis]

I took a look through this and left some suggestions / personal preferences. @stapelberg I don't mean to step on your toes. Just thought those comments might be helpful.

[anthonyrisinger]

Let me know if there's any other changes—I simplify tests with a round-trip through both encode/decode paths.

[anthonyrisinger]

Should be all set! Validated with a Docker container:

$ docker run --rm --privileged -v ~/devel/nftables:/work -w /work golang:1.24-bookworm bash -lc '/usr/local/go/bin/go test -count=5 -run "^TestSetByteOrderRoundTrip$" -run_system_tests .'

go: downloading github.com/mdlayher/netlink v1.8.1-0.20251028132421-dcc6cab9a6eb
go: downloading golang.org/x/sys v0.35.0
go: downloading github.com/vishvananda/netns v0.0.4
go: downloading golang.org/x/net v0.43.0
go: downloading github.com/mdlayher/socket v0.5.1
go: downloading golang.org/x/sync v0.6.0
PASS
ok  	github.com/google/nftables	0.022s

[nickgarlis]

LGTM

[anthonyrisinger]

No real rush on my end (I'm using a private fork at the moment), but is it reasonable to believe this will find its way into the next release, whenever that may be? Mostly looking to check any boxes before I file this away to the back of my mind ☺️