x/net/http2: server unconditionally sends SETTINGS_NO_RFC7540_PRIORITIES=1 by default since v0.50.0, incompatible with connection multiplexers

[lukeo3o1]

Go version

go1.26.0 windows/amd64

Output of go env in your module/workspace:

N/A

What did you do?

Used golang.org/x/net@v0.50.0 (or later) with an HTTP/2 server behind a connection multiplexer such as cmux, which sniffs the connection preface and optionally sends a bare SETTINGS frame before handing the net.Conn off to http2.Server.ServeConn.

What did you see happen?

Clients (Chrome, Edge, etc.) close the connection immediately with:

GOAWAY ErrCode=PROTOCOL_ERROR
Debug="spdy::SETTINGS_DEPRECATE_HTTP2_PRIORITIES value changed after first SETTINGS frame."

What did you expect to see?

http2.Server should not emit SETTINGS_NO_RFC7540_PRIORITIES=1 by default. Prior to v0.50.0, this setting was never sent, and the behavior change should require an explicit opt-in rather than being a side-effect of the default write schedulers (*randomWriteScheduler, *roundRobinWriteScheduler).

---

Root Cause

Two related changes landed in golang.org/x/net@v0.50.0 as part of #75500:

1. A new helper method writeSchedIgnoresRFC7540() was introduced. Notably, both *randomWriteScheduler and *roundRobinWriteScheduler — the two default schedulers most users run — return true:
   https://github.com/golang/net/blob/ebddb99633e0fc35d135f62e9400678492c1d3be/http2/server.go#L660-L671
2. When writeSchedIgnoresRFC7540() returns true, the server now unconditionally appends SettingNoRFC7540Priorities = 1 to its initial SETTINGS frame:
   https://github.com/golang/net/blob/ebddb99633e0fc35d135f62e9400678492c1d3be/http2/server.go#L961-L963

Failure Sequence

1. cmux reads the HTTP/2 client connection preface and sends a bare SETTINGS frame to the client (no SETTINGS_DEPRECATE_HTTP2_PRIORITIES, so value is implicitly 0).
2. The connection is handed off to http2.Server.ServeConn.
3. http2.Server sends its own SETTINGS frame, now including SETTINGS_DEPRECATE_HTTP2_PRIORITIES = 1.
4. The client observes this value changing from 0 to 1 across two SETTINGS frames. RFC 9218 §9.1 prohibits this: a sender MUST NOT revert this setting once enabled.
5. Client closes with GOAWAY / PROTOCOL_ERROR.

Discussion: Why DisableClientPriority Does Not Fix On-wire Regression

DisableClientPriority (planned for Go 1.27, see #75500) is designed to control the server's stream scheduling behavior. It disables client-driven prioritization and selects round-robin across all streams. However, this does not prevent the server from setting SettingNoRFC7540Priorities=1 in its initial SETTINGS frame, as roundRobinWriteScheduler still returns true for writeSchedIgnoresRFC7540().

Therefore, the on-wire protocol regression described above remains unaddressed. There is no current mechanism for applications using cmux-style intermediaries to suppress emission of SETTINGS_NO_RFC7540_PRIORITIES from Go's HTTP/2 server.

---

Possible Directions

1. Restrict writeSchedIgnoresRFC7540() to opt-in schedulers only — Only explicit adoption of the RFC 9218 priority scheduler would emit SETTINGS_NO_RFC7540_PRIORITIES=1. The default scheduling (including round-robin and random) should preserve pre-v0.50.0 on-wire behavior.

2. Add an explicit flag to control emission of SETTINGS_NO_RFC7540_PRIORITIES — e.g., a NoRFC7540Priorities bool field on http2.Server. This would decouple scheduling logic from frame signaling.

Either option would allow advanced use cases (like cmux) to avoid breaking HTTP/2 interoperability with some clients.

Happy to hear thoughts from the maintainers on which direction or solution is preferred.

Related: #75500

[gabyhelp]

Related Issues

• x/net/http2: stream prioritization does not work for small payloads #75936
• x/net/http2: add support for RFC 9218 priorities #75500 (closed)
• x/net/http2: deprecate WriteScheduler #67817
• net/http: bundled http2 uses priority-based scheduler instead of round robin scheduler #64216 (closed)
• proposal: x/net/http2: add "SETTINGS_NO_RFC7540_PRIORITIES" #69917 (closed)
• net/http: Stream starvation in http2 priority write scheduler #58804 (closed)
• net/http: use the http2 priority scheduler by default #18318 (closed)
• net/http: h2 does not support other net.Conn than *tls.Conn #55076 (closed)
• net/http: switch default HTTP/2 write scheduler to round robin [freeze exception] #67706 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[nicholashusin]

Thanks for the report @lukeo3o1 !

Some of my initial thoughts are:

Restrict writeSchedIgnoresRFC7540() to opt-in schedulers only — Only explicit adoption of the RFC 9218 priority scheduler would emit SETTINGS_NO_RFC7540_PRIORITIES=1. The default scheduling (including round-robin and random) should preserve pre-v0.50.0 on-wire behavior.

One thing to note here is that the RFC 9218 priority scheduler will not be an explicit opt-in in Go 1.27. Instead, it will be the default scheduler (although, when no priority is sent by the client, it will behave in a round-robin manner just like the current default round-robin scheduler).

If we do make sending SETTINGS_NO_RFC7540_PRIORITIES customizable, I think we should not tie it to the write scheduler as write schedulers are something that we want to deprecate soon (#67817). As you rightly pointed out, DisableClientPriority also does not prevent SETTINGS_NO_RFC7540_PRIORITIES from being sent. I think this is fine. Setting DisableClientPriority to true explicitly and finding that it results in SETTINGS_NO_RFC7540_PRIORITIES not being sent probably is more confusing for users; one would expect the opposite from the naming.

Add an explicit flag to control emission of SETTINGS_NO_RFC7540_PRIORITIES — e.g., a NoRFC7540Priorities bool field on http2.Server. This would decouple scheduling logic from frame signaling.

I'd personally want to avoid expanding the API surface of http2.Server for a detail that should be transparent for and would not affect most users. If we do add the option to not send SETTINGS_NO_RFC7540_PRIORITIES, I think putting it behind a GODEBUG would be the best choice.

---

Having said all that, I think we first need to evaluate whether not sending SETTINGS_NO_RFC7540_PRIORITIES should be something we support in the first place.

It is unfortunate that we are breaking cmux here, but I'd personally argue that this is working as intended. One should not inject part of a protocol like that and expect things to work okay (of course, I say this with a hindsight bias, and the cmux authors wrote it before RFC 9218 was even published). Our change is in compliance with the RFC, benefits the majority of users, and does not change a promised behavior. It seems to me that this issue should be something that's instead handled by connection mux implementations.

Of course, Hyrum's Law exists, and we're definitely open to revert some behavior here if it's causing a wide breakage. However, I'd like to see more proof of a wide breakage before we do this.

From looking around online, I see that:

• nghttpd always sends SETTINGS_NO_RFC7540_PRIORITIES in their SETTINGS frame, and that seems to be fine.
• lighttpd sends SETTINGS_NO_RFC7540_PRIORITIES too.
• I assume the number of servers doing the same will only increase over time.

I also looked into some projects that depend on cmux. The most notable one I found was etcd-io/etcd#18032, but they seemed to have updated to golang.org/x/net@v0.51.0 without issue (from a quick grep, it seems like they just rely on cmux for gRPC?)

[lukeo3o1]

Thank you for the thorough response, @nicholashusin!

I did notice the upcoming deprecation of WriteScheduler (#67817), and I agree that adding a new API field to http2.Server just for this purpose would be an unnecessary expansion of surface area. A GODEBUG flag does seem like a reasonable middle ground — though I also noticed there is an active proposal (76163) discussing a policy for eventually removing GODEBUG flags. I'm not certain whether that would affect any flag introduced here, but regardless, I wanted to lay out my concrete use case.

Use case

My scenario is simply running HTTP/1.1, HTTP/2, and gRPC all on the same port to reduce deployment overhead. The impact is limited to my own product and does not represent wide-scale breakage.

The background:

I found that once a listener obtained from cmux is passed into http.Server, connections fail to reach the HTTP/2 handler and instead fall back to HTTP/1.1. This was previously reported in #55076 (closed as not planned) and a fix was attempted in #55077. The root cause is that http.Server only activates HTTP/2 when the net.Conn is exactly a *tls.Conn — so connections wrapped by cmux, even when the underlying connection is TLS, fail the type assertion and silently fall back to HTTP/1.1. As a result, cmux.HTTP2() matching effectively never reaches the http2.Server.

After neild suggested using http2.Server.ServeConn directly as an alternative approach, I extended that idea and implemented a fix on the cmux side in soheilhy/cmux#96.

After encountering this, one option I considered was simply dropping support for standard HTTP/2 connections (letting browsers fall back to HTTP/1.1), but that would obviously forfeit the benefits of HTTP/2.

Current thinking

Given your points and the observation that SETTINGS_NO_RFC7540_PRIORITIES=1 is increasingly becoming a de facto standard (adopted by nghttpd, lighttpd, etc.), I think always sending SETTINGS_NO_RFC7540_PRIORITIES=1 is a reasonable direction — the ecosystem does need to move forward.

With that in mind, I realized I could address the cmux compatibility issue upstream: by injecting SETTINGS_NO_RFC7540_PRIORITIES=1 into the initial SETTINGS frame that cmux intercepts, the mux layer would become aware of this behavior from the start. I've addressed this in soheilhy/cmux@b27b6a0, which adds HTTP2MatchHeaderFieldSendSettings to allow callers to pass custom write settings enabling SETTINGS_NO_RFC7540_PRIORITIES=1 to be injected at the matching stage.

However, this approach relies on a guarantee (or at least a stable signal) that Go's HTTP/2 implementation will always send SETTINGS_NO_RFC7540_PRIORITIES=1 going forward. Specifically, it would be helpful to know:

1. Is there a commitment that SETTINGS_NO_RFC7540_PRIORITIES=1 will always be sent in future versions of golang.org/x/net/http2?
2. Alternatively, would it be feasible to export a public constant representing this setting value so that connection-mux implementations can reliably detect and handle it — even if it adds a small amount of API surface?

That said, a regression fix is no longer necessary for my case given the workaround I've found. I mostly wanted to share this context and ask the above questions.

[neild]

To be clear, cmux is definitely broken. You can't just stick additional frames into an HTTP/2 connection. We also aren't going to commit to the contents of the SETTINGS frames we send in the future.

Why is cmux sending a redundant SETTINGS frame, when the server is going send one of its own?

[nicholashusin]

Thanks for understanding @lukeo3o1 ! It's never a happy occasion to break users, but unfortunately completely avoiding breakage is often not possible if we want to make significant change that's ultimately better for most.

As Damien said, we cannot really commit to the contents of the SETTINGS frame. Similarly, I doubt that we'd want to add an exported public constant for this (which, due to Go's compatibility promise, will have to stick around forever).

---

Why is cmux sending a redundant SETTINGS frame, when the server is going send one of its own?

From the docs, it looks like it's mostly to be able to get the HEADERS frame (for matching purpose) from clients who won't send it until they have received a SETTINGS frame from the server?

I'm not sure how common this is in the wild though. Do you see a lot of issues with just using HTTP2HeaderFieldPrefix @lukeo3o1 ?

---

Alternatively—and I want to be clear that this is a hack and not good engineering by any means—if you really want to stick with your current approach, but are afraid that we might change the content of our SETTINGS frame, you can try sniffing our server's SETTINGS frame at startup?

So, before you set up your cmux, create an HTTP/2 server, send a request to it, and see what values are given in the SETTINGS frame. Then, you can pass the parameters in the SETTINGS frame to your modified version of cmux. This way, even if the content of SETTINGS frame changes in the future, your program won't break.

Again, this is a hack, and I'm sure this will eventually break again for some other reasons. Ultimately, the correct approach is to stop injecting frames. But, if you're doing this mostly for a toy project / home lab / etc., this might be "good enough"?

[lukeo3o1]

Thanks again, @nicholashusin and @neild.

I understand your positions clearly. Your direct and candid comments made me realize I should properly evaluate the risks rather than just leaving the hack in place.

Why is cmux sending a redundant SETTINGS frame, when the server is going send one of its own?

The reason it exists is that some gRPC clients won't send their first HEADERS frame until they've received a server SETTINGS frame, which means cmux can't read enough data to distinguish gRPC from standard HTTP/2. That said, in my environment I don't think I actually have any clients that behave this way, so I'll probably just switch to HTTP2HeaderFieldPrefix (as @nicholashusin suggested) and drop the SETTINGS injection entirely.

Just to clarify — this isn't a toy project or home lab; it's a product deployed in multiple production environments, though not publicly exposed.

No action needed on your end — thanks for taking the time 🙏.

[nicholashusin]

That said, in my environment I don't think I actually have any clients that behave this way, so I'll probably just switch to HTTP2HeaderFieldPrefix

Great! :-)