deps: replace unmaintained github.com/soheilhy/cmux

[xw19]

Summary

CRI-O depends on github.com/soheilhy/cmux (v0.1.5, released 2021-02-05) in cmd/crio/main.go to multiplex gRPC and HTTP traffic on a single Unix socket. The library has not received a commit in 5 years, has 26 open issues and 10 open PRs (oldest from 2019) with no maintainer response, and pins a stale golang.org/x/net snapshot from Nov 2020.

This proposes removing it.

Current usage

// cmd/crio/main.go
m := cmux.New(lis)
grpcL := m.MatchWithWriters(cmux.HTTP2MatchHeaderFieldSendSettings("content-type", "application/grpc"))
httpL := m.Match(cmux.HTTP1Fast())

go grpcServer.Serve(grpcL)   // CRI v1 Runtime + Image
go httpServer.Serve(httpL)   // /info, /config, /containers/:id, /pause/:id, /unpause/:id, /debug/*
go m.Serve()

The metrics server and pprof endpoints already use independent listeners.

Proposal

Option A (recommended): remove cmux, split into two listeners.

• gRPC keeps /var/run/crio/crio.sock (existing listen key).
• HTTP inspect API gets its own socket via a new inspect_listen key. Default it to crio.sock initially with a deprecation notice so existing tooling keeps working, then flip the default to a separate socket in a later release.

This matches the design containerd already uses (separate sockets for gRPC, debug, metrics) and the pattern CRI-O's own metrics server uses. Upstream grpc-go explicitly recommends this approach for HTTP-on-the-side endpoints.

Option B (fallback): swap to [elastic/gmux](https://github.com/elastic/gmux). Actively maintained (last release Apr 2026), modern Go and golang.org/x/net, designed specifically for the gRPC+HTTP case. Small refactor in cmd/crio/main.go. Keeps the single-socket behaviour.

Full evaluation of these and other options (cockroachdb/cmux, grpc.ServeHTTP, connect-go) in the design doc linked below.

Acceptance criteria

• github.com/soheilhy/cmux removed from go.mod / vendor
• gRPC (CRI v1) and HTTP inspect endpoints continue to function
• Backwards-compatible default for any new config key
• Existing socat-based integration test (/info through the socket) updated
• docs/ / man pages updated if socket configuration changes

References

• Peer-project parallel: [etcd-io/etcd#18032](Evaluate & potentially replace https://github.com/soheilhy/cmux etcd-io/etcd#18032) — etcd evaluating the same migration since May 2024
• Upstream cmux: https://github.com/soheilhy/cmux (last commit 2021-02-05)

[xw19]

Replacing soheilhy/cmux in CRI-O — Preliminary research

Co-authored-by: Claude Opus 4.7 noreply@anthropic.com

Context

CRI-O uses github.com/soheilhy/cmux v0.1.5 in cmd/crio/main.go to demultiplex two protocols on one Unix domain socket:

• gRPC — the kubelet's CRI v1 Runtime and Image services.
• HTTP/1.1 — an inspect API at /info, /config, /containers/:id, /pause/:id, /unpause/:id, and the /debug/* pprof endpoints.

Both share /var/run/crio/crio.sock, chmod 0o660. The metrics server and a separately-flagged pprof socket already live on independent listeners (see server/metrics/metrics.go), so the multiplexer in main.go is the only remaining cmux dependency.

Part 1 — Why replace it

1.1 Upstream is unmaintained

Metric	Value	Source
Last commit on master	5ec6847 ("Adopt go modules"), 2021-02-05	git log of repo
Last tagged release	v0.1.5 — same commit	git tag
Open issues	26	https://github.com/soheilhy/cmux/issues
Open PRs	10	https://github.com/soheilhy/cmux/pulls

Commit volume by year: 30 (2015), 61 (2016), 18 (2017), 6 (2018), 1 (2019), 3 (2020), 3 (2021), 0 since Feb 2021. Activity collapsed in 2017–18 and stopped entirely in early 2021.

Open PRs include substantive contributions left for years with no maintainer response:

PR	Opened	Title
#104	Mar 2025	chore: Update some dependencies
#100	Oct 2023	fix: typo
#97	Mar 2023	Feature support proxy protocol
#96	Sep 2022	feat: h2 server support
#93	Apr 2022	optimize(matchers): ReadLine block
#92	Feb 2022	do not propagate "use of closed conn" error if expected
#90	Jan 2022	Use new gRPC matcher for README example
#71	Nov 2019	remove support for deprecated tls.VersionSSL30
#66	Mar 2019	README: Update limitations for gRPC clients

PR #66 has sat unmerged for nearly 7 years. PR #104 — "Update some dependencies" — is exactly the maintenance work that would resolve the issues in §1.2, and it has been ignored.

Open issues directly relevant to CRI-O's usage:

• #101 "Upgrade the x/net version in dependencies?" (Nov 2023, ignored)
• #89 "Close not working properly on gRPC graceful stop" (Jul 2021) — CRI-O calls grpcServer.GracefulStop() in its shutdown path
• #99 "HTTP1Fast matcher omits PATCH" — HTTP1Fast is the matcher CRI-O uses; this is a real correctness bug
• #103 "cmux is not usable with GRPC+TLS"

1.2 Stale transitive golang.org/x/net and scanner noise

cmux's go.mod:

require golang.org/x/net v0.0.0-20201202161906-c7110b5ffcbb

That snapshot predates several CVE fixes in golang.org/x/net/http2:

CVE	Fixed in	Impact
CVE-2022-41723 (GHSA-vvpx-j8f3-3w6h, CVSS 7.5)	x/net v0.7.0	HPACK quadratic-time DoS
CVE-2023-39325 (Rapid Reset)	x/net v0.17.0	HTTP/2 stream-reset DoS
CVE-2023-45288 (CONTINUATION flood)	x/net v0.23.0	Header parsing DoS
CVE-2024-45338	x/net v0.33.0	Non-linear HTML parsing

In practice CRI-O's top-level go.mod has a newer golang.org/x/net, so Go's minimum-version-selection picks the fixed code in the compiled binary. The issue is not necessarily exploitable, but:

• go.mod / go.sum still record cmux's pin, so SBOM/CVE scanners (Trivy, Grype, Snyk, AWS Inspector, Red Hat product-security) flag CRI-O.
• Helm hit this exact pattern in helm/helm#11850 and worked around it with go mod edit --replace golang.org/x/net=golang.org/x/net@v0.7.0.
• Kubernetes ingress-nginx hit the same scanner finding in kubernetes/ingress-nginx#9818.
• Because cmux upstream is dead, the only paths to a clean SBOM are: a replace directive in CRI-O's go.mod (suppression, not fix), or removing the dependency.

1.3 Code-quality smells in v0.1.5

Minor on their own but illustrate the lack of upkeep:

• matchers.go references tls.VersionSSL30 (deprecated since Go 1.13; PR Remove unnecessary sleep from tests #71 to remove it unmerged since Nov 2019).
• matchers.go uses ioutil.Discard (io/ioutil deprecated since Go 1.16).
• cmux.go::handleErr calls net.Error.Temporary() (deprecated since Go 1.18, golang/go#45729; staticcheck SA1019).
• CRI-O does not call m.SetReadTimeout(...), so a socket-group member that opens the socket and writes a partial HTTP/2 client preface holds a CRI-O goroutine indefinitely.
• README documents http.Request.TLS is always nil through cmux (doesn't affect CRI-O today — Unix socket, no TLS — but is a known design wart).

1.4 The architectural justification for multiplexing doesn't apply to CRI-O

cmux exists to share one TCP port between multiple protocols — one firewall rule, one TLS termination, one DNS record. None of these apply to CRI-O's setup:

• There is no port — the listener is a filesystem path.
• There is no firewall or middlebox between CRI-O and its clients (kubelet, crictl).
• There is no TLS termination to share.
• CRI-O already runs the metrics server on a separate listener, so the "one socket only" property isn't preserved anyway.

CRI-O is paying the maintenance cost of an unmaintained dependency whose value proposition does not apply to its deployment model.

1.5 Peer-project parallel: etcd

etcd has the same dependency and an open issue evaluating exactly the same migration: etcd-io/etcd#18032, opened May 2024 by an etcd maintainer.

The framing in the etcd issue is nearly identical to this one. The maintainer discussion concluded:

• grpc.Server.ServeHTTP is not a viable replacement — it's still experimental and missing features (referenced grpc/grpc-go#7261).
• Separate listeners is the right answer.
• The blocker is backwards-compatibility for existing users of multiplexed endpoints.

That is the exact trade-off CRI-O faces, and the conclusion CRI-O can build on rather than rediscover.

Part 2 — Alternatives evaluation

CRI-O's actual need is narrow: route a connection on a Unix socket to either a gRPC server or a small HTTP inspect API. No TLS, no other protocols. Options ranked by fit for that specific shape.

#	Option	Maintained	Fit	Verdict
1	Two listeners, no library	n/a — stdlib	★★★★★	Recommended
2	grpc.Server.ServeHTTP + content-type router	Part of grpc-go, marked EXPERIMENTAL	★★	Avoid
3	elastic/gmux	Active (Apr 2026)	★★★★	Best fallback
4	cockroachdb/cmux	Semi-internal	★★★	Stopgap only
5	connect-go	Active, larger change	★★	Out of scope
6	Custom byte-prefix demux	Own code	★★★	Re-implements cmux

2.1 Two separate listeners — recommended

grpcLis, err := server.Listen("unix", config.Listen)
httpLis, err := server.Listen("unix", config.InspectListen)
go grpcServer.Serve(grpcLis)
go httpServer.Serve(httpLis)

Pros

• Zero new dependencies.
• Each socket can have independent permissions — the inspect/debug endpoints can be locked down separately from the CRI gRPC socket. Today anyone who reaches crio.sock can hit /debug/pprof/*.
• Independent shutdown, timeout, and observability per server.
• Matches containerd (containerd-config.toml docs): separate sockets for gRPC, debug, metrics.
• Matches CRI-O's existing metrics-server pattern.
• Upstream grpc-go's explicit recommendation: "If you want an http server for other application like pprof, you can use a different port." — grpc/grpc-go#7261

Cons

• External tooling that hits HTTP endpoints on crio.sock directly would need to switch sockets.
• Requires a config migration: introduce inspect_listen. Mitigation: serve the HTTP endpoints on the gRPC socket for one release with a deprecation log line, then flip the default.

2.2 grpc.Server.ServeHTTP with content-type router

handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    if r.ProtoMajor == 2 && strings.HasPrefix(r.Header.Get("Content-Type"), "application/grpc") {
        grpcServer.ServeHTTP(w, r) // EXPERIMENTAL
    } else {
        infoMux.ServeHTTP(w, r)
    }
})

Cons (significant)

• grpc.Server.ServeHTTP is still marked EXPERIMENTAL (grpc-go server.go).
• Routes through serverHandlerTransport, which grpc-go maintainers call "an experimental adaptor … missing many features including flow control" (grpc-go#7261).
• grpc.Server transport-level options (keep-alives, max-connection-age, window auto-tune) are silently ignored.
• Roughly 2× slower than native gRPC in published benchmarks (emcfarlane.com 2023-05-15: 73 µs/op vs 37 µs/op for BenchmarkGRPC_GetBook).
• History of correctness bugs (grpc-go#1384, grpc-go#3713).

Not appropriate for a kubelet-critical gRPC path.

2.3 elastic/gmux

s := &http.Server{Handler: infoMux}
grpcLis, _ := gmux.ConfigureServer(s, nil)
go grpcServer.Serve(grpcLis)
s.Serve(lis)

Property	Value
Last release	v0.3.3, Apr 2026
go.mod	go 1.24.0, golang.org/x/net v0.50.0, google.golang.org/grpc v1.80.0
License	Apache 2.0
Stars / forks	19 / 9
Source size	~400 lines (mux.go) + ~120 lines (conn.go)

Pros

• Actively maintained by Elastic.
• Modern transitive deps — closes the SBOM noise.
• Designed specifically for the gRPC+HTTP case (doesn't try to demux SSH, raw TCP RPC, etc.).
• Native gRPC server is preserved (no ServeHTTP pitfalls from §2.2).
• Small, auditable codebase.
• Conceptually drop-in for the current cmux usage.

Cons

• Small project; one organisation effectively the sole maintainer. Better than soheilhy/cmux, but bus-factor risk remains.
• Still a multiplexer dependency, so doesn't address the §1.4 architectural objection.

2.4 cockroachdb/cmux

Fork of soheilhy/cmux with patches for CockroachDB's needs. Import-path swap, same API.

Pros

• Minimal code change.

Cons

• Maintained against CockroachDB's needs, not the public's. Its own go.mod may carry stale transitive deps; would need verification.
• Prometheus migrated from cockroachdb/cmux back to soheilhy/cmux in April 2019 (commit 81c42480) because the fork lagged on a needed gRPC fix — illustrates the bus-factor risk.
• Trades one unmaintained dep for a semi-maintained one. Doesn't address §1.4.

2.5 connect-go

Replace grpc.Server with connectrpc.com/connect, whose handlers are plain http.Handler and compose naturally with other HTTP handlers — no multiplexer needed.

Cons

• Much larger change. Requires regenerating service stubs with protoc-gen-connect-go, or dual-registering handlers.
• Adds a (large, active) dependency to remove a (small, dead) one. Out of scope for "replace cmux."

2.6 Custom byte-prefix demux

~50 LoC that peeks the first 24 bytes; HTTP/2 client preface → gRPC, else → HTTP/1.

Cons

• Becomes our own connection multiplexer to maintain.
• Edge cases (buffered-reader handoff so the real server reads the same bytes, SETTINGS-frame handling for Java gRPC clients) are non-trivial.
• Strictly worse than §2.3 unless §2.3 is rejected for some unstated reason.

Recommendation

In priority order:

1. Option 2.1 (two listeners). Best engineering outcome. Matches containerd, matches CRI-O's metrics-server pattern, has upstream grpc-go's explicit blessing, and aligns with where etcd is heading on the same dependency. The backwards-compatibility cost is real but bounded (one deprecation cycle).

2. Option 2.3 (elastic/gmux) if single-socket compatibility must be preserved indefinitely. Modern code, modern deps, narrow scope.

3. Option 2.4 (cockroachdb/cmux) only as a short-term stopgap if neither of the above can ship quickly. Removes the SBOM noise, leaves the architectural and bus-factor problems in place.

Options 2.2, 2.5, 2.6 not recommended unless additional constraints emerge.

Test coverage

Current test coverage of the cmux-multiplexed path is thin:

• Unit tests for inspect handlers call ServeHTTP() directly, bypassing cmux entirely.
• The only integration coverage that exercises HTTP through the real socket is a socat-based test hitting /info.

Any migration should add integration coverage for the HTTP inspect API alongside the change.

Acceptance criteria

• github.com/soheilhy/cmux removed from go.mod / vendor.
• gRPC (CRI v1) Runtime and Image services continue to function.
• HTTP inspect endpoints (/info, /config, /containers/:id, /pause/:id, /unpause/:id, /debug/*) continue to function.
• Backwards-compatible default for any new config key (e.g. inspect_listen defaulting to listen for one release).
• Existing socat-based integration test updated to the new path.
• dependencies.yaml updated if tool versions change.
• docs/ and man pages updated if socket configuration changes.

References

Topic	Link
Upstream cmux	https://github.com/soheilhy/cmux
Last cmux commit	soheilhy/cmux@5ec6847
cmux #101 (ignored x/net upgrade)	soheilhy/cmux#101
etcd parallel issue	etcd-io/etcd#18032
grpc-go: use a different port for HTTP	grpc/grpc-go#7261
containerd config (separate sockets)	https://github.com/containerd/containerd/blob/main/docs/man/containerd-config.toml.5.md
elastic/gmux	https://github.com/elastic/gmux
cockroachdb/cmux	https://github.com/cockroachdb/cmux
connect-go	https://github.com/connectrpc/connect-go
CVE-2022-41723 / GHSA-vvpx-j8f3-3w6h	GHSA-vvpx-j8f3-3w6h
Helm precedent for x/net replace	helm/helm#11850
ServeHTTP performance benchmark	https://www.emcfarlane.com/blog/2023-05-15-grpc-servehttp

[xw19]

/kind feature

[xw19]

After doing a bit more research on why CRI-O uses a single socket, two reasons stood out to me:

1. OpenShift runs strictly on Red Hat Enterprise Linux CoreOS (RHCOS) with aggressive SELinux enforcing policies, especially around Multi-Category Security (MCS). Every file, socket, and port on the host needs to be labeled, tracked, and tightly controlled through policy.

2. The oc debug node paradigm (without host networking)

Because OpenShift worker nodes are effectively immutable, administrators cannot simply SSH into a node and run curl against a local TCP port to debug a failing runtime.

Instead, OpenShift relies on oc debug node, which launches a privileged diagnostic container on the target node.

To debug the runtime, that container needs to communicate with CRI-O.

If the inspect API were exposed on a TCP port (for example 127.0.0.1:6060), the diagnostic container would need hostNetwork: true to access it. In a multi-tenant cluster, that becomes a fairly large security concern.

By exposing the HTTP inspect API over the same UNIX socket using cmux, OpenShift can simply volume-mount the single .sock file into the diagnostic container. This gives administrators HTTP debugging access without requiring access to the host network namespace.

The tradeoff, of course, is slightly higher CPU overhead on every gRPC request due to multiplexing.

So from my perspective, this becomes more of an architectural tradeoff than a purely technical correctness issue. I’d be happy to work on whichever direction the maintainers think makes the most sense.

[xw19]

Also, maybe another reason is why a single socket is in early days this is to maintain compatibility with docker.

[xw19]

@cri-o/cri-o-maintainers I would love to hear your perspective.

[bitoku]

I'm open to having two listeners, though we need to identify what depends on the APIs first.

The debug pod should be almost same as root privilege on the node, so I'm not sure how dangerous having hostNetwork: true is.
And I wonder why we should expose the endpoint via TCP socket.