Kestrel can deadlock registering a certificate file watch (CertificatePathWatcher._metadataLock vs CompositeChangeToken._callbackLock) #70027

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

With a certificate configured by path (Kestrel:Certificates:Default:Path), Kestrel can deadlock while it registers the file watch: at start (BindAsync), or on a configuration reload that points the certificate at a new directory (RebindAsync). On macOS no file has to change. A start that deadlocks never binds. A rebind that deadlocks holds Kestrel's bind semaphore, so later rebinds and StopAsync wait forever.

Expected Behavior

Kestrel starts, or rebinds, with the certificate, whatever thread the file watcher's timer runs on. If a change to the certificate file arrives while the watch is being registered, Kestrel either picks it up or ignores it, and nothing hangs.

Steps To Reproduce

Repro project: https://github.com/BrianWhy/kestrel-cert-watch-deadlock

It is the ASP.NET Core Empty template (dotnet new web) with only Program.cs changed. Program.cs writes a self-signed certificate to a temporary directory once and never changes it. It then starts and stops Kestrel in a loop, with that certificate configured by path (Kestrel:Certificates:Default:Path).

1. dotnet run -c Release
2. When it prints Start <n> hung. Stacks: dotnet-stack report -p <pid>, run that command in another terminal (dotnet tool install -g dotnet-stack).

On ASP.NET Core 10.0.13 / .NET 10.0.13, macOS 27.0.1 arm64, 14 cores, it hung at start 872 after 100 s, and at start 3,604 after 6 min 35 s. On 10.0.12 the same project hung at start 1,930 after about 4 minutes and at start 4,456 after about 10 minutes.

// Starts and stops Kestrel in a loop with a certificate configured by path. The file is written once and never changed.
using System.Security.Cryptography;
using System.Security.Cryptography.X509Certificates;

var pfx = Path.Combine(Directory.CreateTempSubdirectory().FullName, "cert.pfx");
File.WriteAllBytes(pfx, new CertificateRequest("CN=localhost", RSA.Create(2048), HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1)
    .CreateSelfSigned(DateTimeOffset.Now, DateTimeOffset.Now.AddDays(1)).Export(X509ContentType.Pfx));

for (int n = 1; ; n++)
{
    var builder = WebApplication.CreateBuilder(["--urls=https://127.0.0.1:0", $"--Kestrel:Certificates:Default:Path={pfx}"]);
    builder.Logging.ClearProviders();
    var app = builder.Build();
    if (!Task.Run(async () => { await app.StartAsync(); await app.StopAsync(); }).Wait(TimeSpan.FromSeconds(10)))
    {
        Console.WriteLine($"Start {n} hung. Stacks: dotnet-stack report -p {Environment.ProcessId}");
        Console.ReadLine();
        return;
    }
    await app.DisposeAsync();
}

Exceptions (if any)

None: the start hangs. These are the two deadlocked threads, from dotnet-stack report against the repro on 10.0.13 at the hang. Two samples 6 s apart showed the same two threads.

thread-pool thread: PhysicalFilesWatcher's polling timer
  System.Threading.Lock.TryEnterSlow                                    <- waits for _metadataLock
  Kestrel.Core.Internal.CertificatePathWatcher.OnChange(String)
  Primitives.ChangeToken+ChangeTokenRegistration`1.OnChangeTokenFired()
  Primitives.ChangeToken+ChangeTokenRegistration`1+<>c.<RegisterChangeTokenCallback>b__7_0(Object)
  System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean)
  Primitives.CompositeChangeToken.OnChange(Object)                      <- holds _callbackLock
  System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean)
  FileProviders.Physical.PhysicalFilesWatcher.RaiseChangeEvents(Object)
  System.Threading.TimerQueue.FireNextTimers()

thread-pool thread: the host starting
  System.Threading.Monitor.Enter_Slowpath(Object)                       <- waits for _callbackLock
  Primitives.CompositeChangeToken.OnChange(Object)
  Primitives.CompositeChangeToken.get_HasChanged()
  Kestrel.Core.Internal.CertificatePathWatcher.AddWatchUnsynchronized(CertificateConfig)
  Kestrel.Core.Internal.CertificatePathWatcher.UpdateWatches(List`1, List`1)   <- holds _metadataLock
  Kestrel.KestrelConfigurationLoader.Reload()
  Kestrel.Core.KestrelServerImpl+<BindAsync>d__31.MoveNext()
  Kestrel.Core.KestrelServerImpl+<StartAsync>d__28`1.MoveNext()
  Hosting.GenericWebHostService+<StartAsync>d__40.MoveNext()

(ChangeToken.OnChange and RegisterChangeTokenCallback are inlined between get_HasChanged and AddWatchUnsynchronized.)

.NET Version

10.0.402 (SDK); runtime 10.0.13

Anything else?

• ASP.NET Core version: 10.0.13 (Microsoft.AspNetCore.App 10.0.13, host commit 4099f83977)
• IDE: none; built and run with the dotnet CLI
• dotnet --info (abridged): SDK 10.0.402, Host 10.0.13 arm64, OS Mac OS X 27.0.1, RID osx-arm64

A longer test program that moves the certificate between two directories on every configuration reload hung the same way on 10.0.13, entering through KestrelServerImpl.RebindAsync, at reload 11,018.