.NET 8 WebApi Experiences a TLS Handshake Error when running in a ubi8 FIPS-enabled container

[KenMcC98]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

When we run containerised .NET 8 WebApis (both custom and starter templates) over HTTPS in our OpenShift Cluster, which is FIPS-enabled, we experience a handshake error when trying to connect using TLSv1.3.

Example error we receive over CURL from a base WebApi template:

$ curl -kv https://<openshift-url>/weatherforecast
* processing: https://<openshift-url>/weatherforecast
*   Trying <ip>:443...
* Connected to <openshift-url> (::1) port 443
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS alert, no application protocol (632):
* OpenSSL/3.1.1: error:0A000460:SSL routines::reason(1120)
* Closing connection
curl: (35) OpenSSL/3.1.1: error:0A000460:SSL routines::reason(1120)

The error only occurs when the container is in FIPS mode, we can successfully host and connect to it on our local desktops (which aren't FIPS enabled), e.g.

$ curl -kv https://localhost:8081/weatherforecast
* processing: https://localhost:8081/weatherforecast
*   Trying [::1]:8081...
* Connected to localhost (::1) port 8081
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  <certificate-details>
*  SSL certificate verify result: self-signed certificate (18), continuing anyway.
* using HTTP/2
* h2 [:method: GET]
* h2 [:scheme: https]
* h2 [:authority: localhost:8081]
* h2 [:path: /weatherforecast]
* h2 [user-agent: curl/8.2.1]
* h2 [accept: */*]
* Using Stream ID: 1
> GET /weatherforecast HTTP/2
> Host: localhost:8081
> User-Agent: curl/8.2.1
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 200 
< content-type: application/json; charset=utf-8
< date: Tue, 05 Mar 2024 10:08:42 GMT
< server: Kestrel
< 
* Connection #0 to host localhost left intact
[{"date":"2024-03-06","temperatureC":25,"temperatureF":76,"summary":"Sweltering"},{"date":"2024-03-07","temperatureC":8,"temperatureF":46,"summary":"Freezing"},{"date":"2024-03-08","temperatureC":8,"temperatureF":46,"summary":"Mild"},{"date":"2024-03-09","temperatureC":41,"temperatureF":105,"summary":"Hot"},{"date":"2024-03-10","temperatureC":52,"temperatureF":125,"summary":"Hot"}]

A similar .NET 6 sample application behaves correctly in both above scenarios.

Expected Behavior

We would expect the connection in the FIPS-enabled environment to be successful, i.e. the same response as the example localhost connection.

If we replace the .NET 8 WebApis with .NET 6 sample WebApis, the handshake will successfully complete in the FIPS OpenShift Cluster.

Steps To Reproduce

I was able to recreate the issue locally through Podman + Podman Desktop.
The code is in this repo: https://github.com/KenMcC98/dotnet-webapp1-container
I've tried to keep it as straightforward and simple as I can, there is a bundled README.md which walks through steps to reproduce, and will require Podman (or Docker with minor filename/command changes).

The repo contains a basic WebApi that targets .NET 6 and 8. To compare the scenarios described, it contains several containerfiles to simulate FIPS and non-FIPS containers which are spun up using a compose file.

FIPS mode is simulated for OpenSSL (which .NET uses) using the OPENSSL_FORCE_FIPS_MODE=1 environment variable. This will enforce FIPS compliance for all RedHat distributions of OpenSSL (UBI8 and Fedora)

The included README.md walks through the steps to recreate, summary of the readme steps:

• create sample certs using bundled sh script
• podman build four images:
  • .net 8 on ubi8
  • .net 6 on ubi8
  • .net 8 on fedora39
  • .net 6 on fedora39
• running with podman-compose up spawns:
  • n8-wa1-ubi8 - .NET 8 on ubi8
    • works without FIPS, error with FIPS
  • n6-wa1-ubi8 - .NET 6 on ubi8
    • works without FIPS, works with FIPS
  • n8-wa1-fedora - .NET 8 on Fedora 39
    • works without FIPS, error with FIPS
  • n6-wa1-fedora - .NET 6 on Fedora 39
    • works without FIPS, works with FIPS
• curl the containers manually:
  • curl -kv https://localhost:<8081|8082|8083|8084|8085>/WeatherForecast

The n8-wa1-*-fips containers will experience the errors, with the others being successful.

Exceptions (if any)

When inspecting the container logs, we find the following exception logged at the debug level when the error occurs:

System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
 ---> Interop+OpenSsl+SslException: SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL.
 ---> Interop+Crypto+OpenSslCryptographicException: error:142320EB:SSL routines:tls_handle_alpn:no application protocol
   --- End of inner exception stack trace ---
   at Interop.OpenSsl.DoSslHandshake(SafeSslHandle context, ReadOnlySpan`1 input, Byte[]& sendBuf, Int32& sendCount)
�   at System.Net.Security.SslStreamPal.HandshakeInternal(SafeDeleteSslContext& context, ReadOnlySpan`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions)
   --- End of inner exception stack trace ---
�   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken)
�   at Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware.OnConnectionAsync(ConnectionContext context)

.NET Version

8.0.102

Anything else?

When recreating the issue locally, I was using a Fedora 39 host machine. I installed podman and podman desktop to manage the containers. It does not require or mandate FIPS on the host, FIPS was simulated using the environment variable OPENSSL_FORCE_FIPS_MODE=1 inside the containers.

OpenSSL links:
OpenSSL FIPS - FIPS Module
RedHat OpenSSL Force-FIPS environment variable - Handling FIPS Mode in Upstream Projects

dotnet --info from inside one of the containers used to recreate the issue:

dotnet --info
.NET SDK:
 Version:           8.0.102
 Commit:            b7800db369
 Workload version:  8.0.100-manifests.03fa9662

Runtime Environment:
 OS Name:     rhel
 OS Version:  8
 OS Platform: Linux
 RID:         rhel.8-x64
 Base Path:   /usr/lib64/dotnet/sdk/8.0.102/

.NET workloads installed:
 Workload version: 8.0.100-manifests.03fa9662
There are no installed workloads to display.

Host:
  Version:      8.0.2
  Architecture: x64
  Commit:       1381d5ebd2

.NET SDKs installed:
  6.0.127 [/usr/lib64/dotnet/sdk]
  8.0.102 [/usr/lib64/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.27 [/usr/lib64/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.2 [/usr/lib64/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.27 [/usr/lib64/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.2 [/usr/lib64/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
  None

Environment variables:
  DOTNET_ROOT       [/usr/lib64/dotnet]

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

[martincostello]

Possibly a duplicate of dotnet/runtime#99289?

[rzikm]

Possibly a duplicate of dotnet/runtime#99289?

It is. Unfortunately, the only workaround is not using TLS 1.3 or not running in FIPS-only mode.