caddy-l4

Upstream referral from OPNsense Caddy maintainer This issue is being submitted here at the explicit recommendation of the OPNsense Caddy plugin maintainer, @Monviech, following the discussion in "opnsense/plugins#5791" (https://github.com/opnsense/plugins/issues/5791#issuecomment-6214345486). After consulting with others, he concluded that this problem would be better addressed upstream in caddy-l4, either through a code-level solution or improved documentation. He specifically indicated that modifying the OPNsense plugin's configuration generation to accommodate this behavior would be undesirable. I am therefore reporting this directly to the upstream project as requested, rather than seeking an OPNsense-specific fix. 1. Environment 1a. Operating system and version OPNsense 26.7 (FreeBSD, amd64) os-caddy plugin 2.2.1_1 Windows client: Google Chrome / Chromium Exact packaged caddy-l4 revision: not yet identified 1b. Caddy version Caddy 2.11.4 2. Description 2a. What happens I have multiple HTTPS hostnames sharing the same public IP and covered by a wildcard certificate. Some hostnames are handled directly by Caddy's HTTP server, while others use caddy-l4 SNI matching through for TLS passthrough without TLS termination. In one Chrome profile, navigating to a TLS-passthrough hostname consistently resulted in a completely blank page. Chrome DevTools showed: The expected upstream application was never reached by that request. The application worked normally in Chrome Incognito and Microsoft Edge. A separate HTTPS/TCP connection returned the expected application HTML without the header. Disabling the browser HTTP cache and bypassing service workers did not resolve the problem. No JavaScript console errors were present. Adding a host-specific HTTP 421 fallback to Caddy's HTTP configuration immediately and completely resolved the problem in the previously affected Chrome profile. The leading explanation is HTTP/2 cross-origin connection coalescing: 1. Chromium establishes an HTTP/2 connection to a hostname handled by Caddy's HTTP server. 2. The certificate also covers another hostname configured for Layer4 TLS passthrough. 3. Chromium reuses the existing HTTP/2 connection for that second hostname. 4. The Layer4 SNI matcher cannot reconsider its routing decision because the TLS handshake has already occurred. 5. The HTTP/2 request reaches Caddy's HTTP handler rather than the intended Layer4 upstream. 6. Caddy responds with HTTP 200 and an empty body. This mechanism was not independently verified with a connection-level trace, but the successful HTTP 421 workaround strongly supports it. 2b. Why it's a bug (if it's not obvious) HTTP/2 connection coalescing is legitimate when the browser considers the existing connection authoritative for multiple origins. However, routing based on TLS SNI cannot distinguish subsequent HTTP/2 requests carrying a different on the same connection. RFC 9113, Section 9.1.1, explicitly describes this interoperability problem and the use of HTTP 421 to reject misdirected requests. https://www.rfc-editor.org/rfc/rfc9113.html#section-9.1.1 Without an appropriate fallback, the request receives an apparently successful empty response, causing a blank page rather than allowing the browser to retry through a connection with the correct SNI. I understand that caddy-l4 operates on connections rather than individual HTTP/2 streams, so it may not be possible to resolve this directly within the Layer4 matcher. Nevertheless, this interaction between caddy-l4 and Caddy's HTTP server seems worth documenting or mitigating. 2c. Log output The relevant evidence came from Chrome DevTools. Failing document response: A separate direct HTTPS connection returned the expected application HTML. Both observed failing document requests used HTTP/2. HTTP/3 was not involved. The workaround passed Caddy configuration validation: After applying the workaround, the previously affected browser profile successfully loaded the application. 2d. Workaround(s) I added an HTTP 421 fallback using an additional Caddyfile imported into the existing configuration: This catches requests for the passthrough hostname when they incorrectly reach Caddy's HTTP handler. Chromium can then retry on a separate connection with the correct SNI, allowing the Layer4 route to select the intended upstream. This workaround completely resolved the observed issue. It required no changes to: Existing Layer4 routes The TLS passthrough upstream Certificates Browser storage Caddy's generated configuration or plugin templates 2e. Relevant links Original downstream report in OPNsense, including the maintainer's recommendation to report this upstream: https://github.com/opnsense/plugins/issues/5791 HTTP/2 specification, Section 9.1.1: https://www.rfc-editor.org/rfc/rfc9113.html#section-9.1.1 I also reviewed some potentially related caddy-l4 reports: #421: Similar blank-page symptoms, ultimately attributed to a QUIC matcher incorrectly configured inside a TCP listener wrapper. #375: Intermittent TLS failures caused by conflicting listeners, resolved through listener wrappers. #328: Empty HTTP 200 responses associated with a port-binding/configuration conflict. These appear distinct from the HTTP/2 connection-coalescing scenario and successful HTTP 421 mitigation described here. 3. Tutorial (minimal steps to reproduce the bug) The observed configuration was generated by the OPNsense Caddy plugin using . The following illustrates the relevant topology; it has not been independently verified as a minimal standalone reproducer: The HTTP-served hostname must have a TLS certificate that also authenticates the passthrough hostname, such as an appropriate wildcard certificate. The observed failure scenario is: 1. Establish an HTTP/2 connection to the hostname handled by Caddy's HTTP server. 2. Navigate to the Layer4 passthrough hostname using the same Chromium profile. 3. If Chromium coalesces the request onto the existing connection, the request may reach Caddy's HTTP server instead of the intended upstream. 4. Observe an empty HTTP 200 response. 5. Add the HTTP 421 fallback shown above. 6. Apply the configuration and repeat the test. On the affected system, step 6 immediately resolved the problem. Because the failure depends on browser connection-pooling decisions, it may not reproduce consistently in a fresh browser or with standalone curl requests. Requested upstream guidance** Would it be appropriate to document this HTTP/2 interoperability caveat for deployments combining , SNI-based TLS passthrough and ordinary HTTPS sites on the same listener? Specifically, should an HTTP 421 fallback for Layer4-exclusive HTTPS hostnames be recommended, or is there a more appropriate solution involving or another Caddy mechanism? If a code-level mitigation is feasible, I would be willing to test it against the affected configuration.

Explain why HTTP/2 clients may send requests for an SNI-passthrough hostname over a connection served by the HTTP app, and how to make them retry with or a per-host 421 response. Closes #479.

Summary Sorry if I've made some noise in the repo these past few days, that wasn't my intention 😅. I'm opening this issue to give an overview of the three issues and PRs I've opened, and to explain why. At OpenVidu we run caddy-l4 as our public TLS entry point: a layer4 server on TCP 443 terminates TLS and proxies each connection either to a TURN server or to Caddy's HTTP server. While investigating connections that piled up on some deployments, we found three places where caddy-l4 keeps a TCP connection with no time limit, so a buggy or hostile client can hold sockets, goroutines and conntrack entries for as long as it wants. To solve it on our side, we use our own fork with the three fixes for now, and we opened one issue and one PR for each problem. They are independent: they merge cleanly in any order. Where a connection can get stuck The lifetime of a connection through a + route, and what bounds each stage: Green: existing limit. Orange: added or fixed by these PRs. Dashed: no limit, by design. Only the matching stage has a limit today, and the routes clear it before the handlers run. 1. TCP FIN after (#465, #466) Problem. When one side finishes, half-closes the other with . Behind the handler, that connection is a , and only sends : as documented, it does not shut down the underlying connection, so no TCP FIN is sent. Anything that only looks at TCP (a layer 4 load balancer, a NAT's conntrack, clients that ignore ) sees the connection as still established, and the proxy keeps waiting for a peer that was never told the server finished. Improvement. After , the proxy also shuts down the writing side of the underlying TCP connection, for the downstream (TLS terminated by the handler) and for TLS upstreams. The peer gets the same FIN it gets over plain TCP. Default if merged. Always on, with no option: it makes TLS behave like plain TCP, which is what means for the proxy. A peer that handles correctly sees no difference. 2. (#467, #468) Problem. Once one side has finished, waits for the other side to finish with no deadline. A client that reads the answer and then neither sends nor closes (or an upstream that does the same after the client has finished) keeps both sockets and the two goroutines indefinitely. The server's only applies to UDP, and TCP keepalive keeps such connections alive. Even with (1), the proxy has no way to give up on a peer that never closes. Improvement. An optional on the handler. It arms when one side has finished, and it is an idle timeout: every read restarts it, so data still flowing in the remaining direction is never cut. On expiry the connection is closed. It is the equivalent of HAProxy's and . Default if merged. , which waits as long as today. Half-closed connections are legitimate (a client can and then wait for a long answer), so cutting them must be a choice. Negative values are rejected. 3. (#469, #470) Problem. The handler runs the handshake with no deadline, and the routes have already cleared the matching deadline. On a route without matchers, or whose matchers do not read the ClientHello (such as ), a client that sends no ClientHello, only part of one, or a complete ClientHello and nothing else keeps the connection forever. On a route matched by a matcher (such as ), the matching timeout covers the first two, but a client that stops right after its ClientHello is still kept forever. Improvement. An optional on the handler. The deadline is set before the handshake and cleared when it completes, so it never applies to the traffic that follows. It is the equivalent of nginx's in its stream module and HAProxy's . Default if merged.** , which keeps the handshake without a limit, as today. Negative values are rejected. Defaults The two new options default to , so merging the three PRs changes nothing for existing configurations except the FIN of (1), which only makes TLS behave like TCP. That keeps each PR easy to review and safe to release. Whether caddy-l4 should eventually ship non-zero defaults is a separate decision, and we would like the maintainers' view on it. Some references: What OpenVidu sets from its next release (3.10.0), where the same port carries HTTPS, WebSocket signalling and TURN over TLS: A browser or a TURN client completes the handshake in well under a second, and once a side has finished, a legitimate peer closes its side within milliseconds, so neither value cut a real connection in our tests on every OpenVidu topology, including calls with long quiet periods. Since is an idle timeout, even a slow answer that keeps sending data is not affected. Personally, I think it would be nice for these options to have non-zero defaults, so that whoever runs into these problems does not have to set them by hand, and the configuration stays as clear and clean as possible. I've always really liked Caddy's defaults :), and I think one of its strengths is precisely how easy its configuration is to read and understand. If non-zero defaults are welcome, values in this range (for example between 10s and 60s, around 30s to 60s) would protect every deployment out of the box. We are happy to send that as a follow-up PR once the options are in. Merge order Any. The three PRs touch different code paths (#466 and #468 both change , but in different places), and every pair merges without conflicts.