net/http: add support for SETTINGS_ENABLE_CONNECT_PROTOCOL

[ethanpailes]

Overview

I propose extending the x/net/http2 and net/http packages to support the extended CONNECT protocol described in RFC 8441. Support for this will be always-on in the http server, and the http client will automatically detect the presence of support based on the settings flag and adjust the checks it applies to outgoing requests to allow the for the extended CONNECT protocol. http.Request will be extended with a new Protocol member to allow users to set the :protocol psudoheader.

Public Interface Changes

http.Request will gain a new member, Protocol of type string.

All other functionality can be achieved with existing interfaces.

Implementation

An example implementation of SETTINGS_ENABLE_CONNECT_PROTOCOL support for both the client and the server can be found on gerrit. This is just meant to be a proof of concept, not a fully production ready implementation. Rather than using a field on http.Request it uses a magic header called HACK-HTTP2-Protocol, but the intention is that this would be switched out for req.Protocol for real code.

The implementation is pretty non-invasive. It consists of extending the code to allow the use of a new :protocol psudo header, sending the new settings flag during the initial handshake, and loosening some checks when the extended CONNECT protocol is in use.

Probably the most complex thing about the implementation I was able to come up with is the fact that a CONNECT request coming from the client can now trigger a ping HTTP/2 frame if the server has not yet sent headers. This is required for an extended CONNECT to work if is the very first request on a given http2.Transport object.

Related Issues

#49918 contains some discussion about some of my initial implementation work and links to further context.

[ianlancetaylor]

CC @neild @bradfitz

[neild]

How do we handle a request sent on an HTTP/1 connection with a Protocol field set? Return an error? Set Upgrade: $PROTOCOL and Connection: upgrade?

An alternative to adding a new field might be to set a :protocol key in the Request.Header map.

[ethanpailes]

How do we handle a request sent on an HTTP/1 connection with a Protocol field set? Return an error? Set Upgrade: $PROTOCOL and Connection: upgrade?

My inclination is to return an error since my understanding of go's philosophy is that it is a one-way-to-do-things language and you can already set the Upgrade and Connection headers in the Request.Header map. I can definitely sympathize with the alternative view that it is less surprising for Protocol = "websocket" to work regardless of the underlying transport though so I'd be happy to do it either way.

An alternative to adding a new field might be to set a :protocol key in the Request.Header map.

I like this approach because it feels a little weird for Protocol to be a top level field on such a common type when it will be so rarely used. I think I actually tried this when implementing support but ran into an error. I decided to propose a new field because psudoheaders are not really supposed to be set by the user, but I'm not so sure how good a justification this is. On reflection, maybe allowing people to set ":protocol" in the headers map by relaxing some checks that forbid user defined psudoheaders is the best way to go since it doesn't shove the feature in users face quite so much. If you're happy with that approach, I would be as well (really I don't have particularly strong feelings in any direction so I'm happy to do whatever, but this design seems good).

[neild]

I don't have a strong feeling on a Protocol field vs. setting a :protocol header. Perhaps @bradfitz has an opinion.

An argument for a Protocol field might be if allows websockets implementations to be agnostic of HTTP/1 or HTTP/2 with net/http handling translation between the Protocol field and the underlying request (pseudo-)headers. I don't know if that's feasible.

We do have existing cases where you can set either a field in the Request or a header (e.g., Content-Length), so doing that for Upgrade and Connection wouldn't necessarily be horrible.

[ethanpailes]

An argument for a Protocol field might be if allows websockets implementations to be agnostic of HTTP/1 or HTTP/2 with net/http handling translation between the Protocol field and the underlying request (pseudo-)headers. I don't know if that's feasible.

The websocket handshake for HTTP/1.1 and HTTP/2 are sufficiently different that attempting this is probably not a good idea. There is a bunch of Sec-* header stuff in HTTP/1.1 that isn't needed in HTTP/2 because psudoheaders are not user-settable from within a browser. The HTTP/2 handshake doesn't need to muck about with nonces for example. Since they are different enough any websocket library will need to explicitly support both protocols unless we move most of the handshake logic into the http and http2 packages themselves, which doesn't seem ideal.

We do have existing cases where you can set either a field in the Request or a header (e.g., Content-Length), so doing that for Upgrade and Connection wouldn't necessarily be horrible.

Gotcha, that is good context and weakens my one-way-to-do-things argument.

I think I'm still a little inclined towards allowing a :protocol header, but not that strongly.

[bradfitz]

I'd rather put :protocol in Request.Header (akin to http.TrailerPrefix) over adding a new Request.Protocol string field.

[neild]

Proposal SGTM with a :protocol pseudo-header in Request.Header.

[ethanpailes]

Should I clean up the gerrit patch to use a :protocol pseudo-header in Request.Header and request another review, or is there another step required for the proposal to be accepted?