[Kestrel] HTTPS with ServerCertificateSelector, build full chain every connection, who to cache it?

[WhitePhoera]

Hello, issue happened with YARP, but code itself is Kestrel actually.
I had implemented dynamic certificate loading(by reload command),

            ServerCertificateSelector = (ctx, sni) =>
            {
                if (!string.IsNullOrWhiteSpace(sni) && KnownCertificates.GetCertificate(sni) is { } cert)
                {
                    return cert;
                }
                ctx.Abort();
                return KnownCertificates.DefaultCertificate;
            }
        });

which worked fine so far with Let's Encrypt certificates and others, where CA is known to system.
But we added other certificate which have it's own CA, which is not known to system.
Under load handshake time increases dramaticaly, by doing dotnet-stack report, i found that most thread where stuck with building full chain:

Thread (0x682D):
  [Native Frames]
  System.Private.CoreLib!System.Threading.ManualResetEventSlim.Wait(int32,value class System.Threading.CancellationToken)
  System.Private.CoreLib!System.Threading.Tasks.Task.SpinThenBlockingWait(int32,value class System.Threading.CancellationToken)
  System.Private.CoreLib!System.Threading.Tasks.Task.InternalWaitCore(int32,value class System.Threading.CancellationToken)
  System.Private.CoreLib!System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(class System.Threading.Tasks.Task)
  System.Net.Http!System.Threading.Tasks.TaskCompletionSourceWithCancellation`1[System.__Canon].WaitWithCancellation(value class System.Threading.CancellationToken)
  System.Net.Http!System.Net.Http.HttpConnectionPool+<GetHttp11ConnectionAsync>d__75.MoveNext()
  System.Private.CoreLib!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start(!!0&)
  System.Net.Http!System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(class System.Net.Http.HttpRequestMessage,bool,value class System.Threading.CancellationToken)
  System.Net.Http!System.Net.Http.HttpConnectionPool+<SendWithVersionDetectionAndRetryAsync>d__83.MoveNext()
  System.Private.CoreLib!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start(!!0&)
  System.Net.Http!System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(class System.Net.Http.HttpRequestMessage,bool,bool,value class System.Threading.CancellationToken)
  System.Net.Http!System.Net.Http.HttpConnectionPoolManager.SendAsyncCore(class System.Net.Http.HttpRequestMessage,class System.Uri,bool,bool,bool,value class System.Threading.CancellationToken)
  System.Net.Http!System.Net.Http.HttpConnectionPoolManager.SendAsync(class System.Net.Http.HttpRequestMessage,bool,bool,value class System.Threading.CancellationToken)
  System.Net.Http!System.Net.Http.HttpConnectionHandler.SendAsync(class System.Net.Http.HttpRequestMessage,bool,value class System.Threading.CancellationToken)
  System.Net.Http!System.Net.Http.HttpMessageHandlerStage.Send(class System.Net.Http.HttpRequestMessage,value class System.Threading.CancellationToken)
  System.Net.Http!System.Net.Http.DiagnosticsHandler.SendAsync(class System.Net.Http.HttpRequestMessage,bool,value class System.Threading.CancellationToken)
  System.Net.Http!System.Net.Http.HttpMessageHandlerStage.Send(class System.Net.Http.HttpRequestMessage,value class System.Threading.CancellationToken)
  System.Net.Http!System.Net.Http.SocketsHttpHandler.Send(class System.Net.Http.HttpRequestMessage,value class System.Threading.CancellationToken)
  System.Net.Http!System.Net.Http.HttpMessageInvoker.Send(class System.Net.Http.HttpRequestMessage,value class System.Threading.CancellationToken)
  System.Net.Http!System.Net.Http.HttpClient.Send(class System.Net.Http.HttpRequestMessage,value class System.Net.Http.HttpCompletionOption,value class System.Threading.CancellationToken)
  System.Net.Http!System.Net.Http.HttpClient.Send(class System.Net.Http.HttpRequestMessage,value class System.Threading.CancellationToken)
  ?!?
  System.Private.CoreLib!System.Reflection.RuntimeMethodInfo.Invoke(class System.Object,value class System.Reflection.BindingFlags,class System.Reflection.Binder,class System.Object[],class System.Globalization.CultureInfo)
  System.Security.Cryptography.X509Certificates!Internal.Cryptography.Pal.CertificateAssetDownloader+<>c__DisplayClass5_0.<CreateDownloadBytesFunc>b__0(class System.String,value class System.Threading.CancellationToken)
  System.Security.Cryptography.X509Certificates!Internal.Cryptography.Pal.CertificateAssetDownloader.DownloadAsset(class System.String,value class System.TimeSpan)
  System.Security.Cryptography.X509Certificates!Internal.Cryptography.Pal.CertificateAssetDownloader.DownloadCertificate(class System.String,value class System.TimeSpan)
  System.Security.Cryptography.X509Certificates!Internal.Cryptography.Pal.OpenSslX509ChainProcessor.FindChainViaAia(class System.Collections.Generic.List`1<class System.Security.Cryptography.X509Certificates.X509Certificate2>&)
  System.Security.Cryptography.X509Certificates!Internal.Cryptography.Pal.ChainPal.BuildChain(bool,class Internal.Cryptography.ICertificatePal,class System.Security.Cryptography.X509Certificates.X509Certificate2Collection,class System.Security.Cryptography.OidCollection,class System.Security.Cryptography.OidCollection,value class System.Security.Cryptography.X509Certificates.X509RevocationMode,value class System.Security.Cryptography.X509Certificates.X509RevocationFlag,class System.Security.Cryptography.X509Certificates.X509Certificate2Collection,value class System.Security.Cryptography.X509Certificates.X509ChainTrustMode,value class System.DateTime,value class System.TimeSpan,bool)
  System.Security.Cryptography.X509Certificates!System.Security.Cryptography.X509Certificates.X509Chain.Build(class System.Security.Cryptography.X509Certificates.X509Certificate2,bool)
  System.Net.Security!System.Net.Security.SslStreamCertificateContext.Create(class System.Security.Cryptography.X509Certificates.X509Certificate2,class System.Security.Cryptography.X509Certificates.X509Certificate2Collection,bool,class System.Net.Security.SslCertificateTrust)
  System.Net.Security!System.Net.Security.SecureChannel.AcquireServerCredentials(unsigned int8[]&)
  System.Net.Security!System.Net.Security.SecureChannel.GenerateToken(value class System.ReadOnlySpan`1<unsigned int8>,unsigned int8[]&)
  System.Net.Security!System.Net.Security.SecureChannel.NextMessage(value class System.ReadOnlySpan`1<unsigned int8>)
  System.Net.Security!System.Net.Security.SslStream.ProcessBlob(int32)
  System.Net.Security!System.Net.Security.SslStream+<ReceiveBlobAsync>d__176`1[System.Net.Security.AsyncReadWriteAdapter].MoveNext()
  System.Private.CoreLib!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start(!!0&)
  System.Net.Security!System.Net.Security.SslStream.ReceiveBlobAsync(!!0)
  System.Net.Security!System.Net.Security.SslStream+<ForceAuthenticationAsync>d__175`1[System.Net.Security.AsyncReadWriteAdapter].MoveNext()
  System.Private.CoreLib!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start(!!0&)
  System.Net.Security!System.Net.Security.SslStream.ProcessAuthenticationAsync(bool,bool,value class System.Threading.CancellationToken)
  System.Net.Security!System.Net.Security.SslStream.AuthenticateAsServerAsync(class System.Net.Security.SslServerAuthenticationOptions,value class System.Threading.CancellationToken)
  Microsoft.AspNetCore.Server.Kestrel.Core!Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware.DoOptionsBasedHandshakeAsync(class Microsoft.AspNetCore.Connections.ConnectionContext,class System.Net.Security.SslStream,class Microsoft.AspNetCore.Server.Kestrel.Core.Internal.TlsConnectionFeature,value class System.Threading.CancellationToken)
  Microsoft.AspNetCore.Server.Kestrel.Core!Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware+<OnConnectionAsync>d__17.MoveNext()
  System.Private.CoreLib!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start(!!0&)
  Microsoft.AspNetCore.Server.Kestrel.Core!Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware.OnConnectionAsync(class Microsoft.AspNetCore.Connections.ConnectionContext)
  Eternalhost.L7Proxy!Program+<>c__DisplayClass0_2+<<<Main>$>b__21>d.MoveNext()
  System.Private.CoreLib!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start(!!0&)
  Eternalhost.L7Proxy!Program+<>c__DisplayClass0_2.<<Main>$>b__21(class Microsoft.AspNetCore.Connections.ConnectionContext)
  Microsoft.AspNetCore.Server.Kestrel.Core!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection`1+<ExecuteAsync>d__6[System.__Canon].MoveNext()
  System.Private.CoreLib!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start(!!0&)
  Microsoft.AspNetCore.Server.Kestrel.Core!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection`1[System.__Canon].ExecuteAsync()
  Microsoft.AspNetCore.Server.Kestrel.Core!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection`1[System.__Canon].System.Threading.IThreadPoolWorkItem.Execute()
  System.Private.CoreLib!System.Threading.ThreadPoolWorkQueue.Dispatch()
  System.Private.CoreLib!System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()

since simple load is handled nicely, but still connection time was bigger in comparison with other domains, i assume that any https-connection is doing that code.
After i added that CA into system, handshake does not hangs any more.

But looks like every connection does building full chain.
Is there is the way to precache it? so i can do that once when i load certificate into memory?

I took stacktrace from .net 7, but .net 6 had same bottleneck.

[davidfowl]

Can you show your KnownCertificates.GetCertificate method?

There's 2 ways to handle this:

1. Build the chain once for this unknown CA (see this for a similar fix)
2. You can use the new SslStreamCertificateContext (this is new in .NET 6)

If you choose 2, then you'll need to use the OnAuthenticate callback instead. Getting the host name is a little more complex though (I believe you can get it from the underlying SSlStream instance).

o.OnAuthenticate = (ctx, options) =>
{
    var sni = ctx.Features.Get<SslStream>()?.TargetHostName;

    if (!string.IsNullOrWhiteSpace(sni) && KnownCertificates.GetCertificate(sni) is { } certContext)
    {
        options.ServerCertificateContext = certContext;
    }
    else
    {
        ctx.Abort();
    }
};

If you're unable to get the host name then we need to look at providing a simpler API to get it.

[WhitePhoera]

Can you show your KnownCertificates.GetCertificate method?
It is dictionary lookup for Dictionary<string,X509Certificate2>, also wildcard lookup as well.
Certificates loaded into memory like:

X509Certificate2.CreateFromPemFile(Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "cert.crt"), Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "cert.key"))

Default loaded at start of app of cource, other certificates loaded at startup and HUP signal.

thanks, i will try those ways.

This issue has been resolved and has not had any activity for 1 day. It will be closed for housekeeping purposes.

See our Issue Management Policies for more information.

[WhitePhoera]

If you're unable to get the host name then we need to look at providing a simpler API to get it.
Trying to use method 2, since it seems to offer better perfomance(i can build context from chain from method 1), but can't get nor SslStream, nor SNI.
tryed both at windows and linux(WSL), second is target platform actually.

@davidfowl , any tips where to look?

[WhitePhoera]

I decided to switch to TlsHandshakeCallbackOptions, where there is no issue with SNI.