Earlier this year I gave a presentation at SUGCON Europe on safe usage of the .NET HttpClient in Sitecore called Stop Worrying and Love the HttpClient. At the end of the presentation I promised that I would write a blog post on the subject. Today I finally make good on that promise.

References

The issues and solutions with the HttpClient that I cover below are well-documented, and I'd like to give a big thanks to resources I've referenced over the years in dealing with these issues.

• Are You Using HttpClient in The Right Way? by Rahul Nath
• Beware of the .NET HttpClient by Nima Ara
• HttpClient Connection Pooling in .NET Core by Steve Gordon
• Singleton HttpClient? Beware of this serious behaviour and how to fix it by Ali Kheyrollahi
• You're using HttpClient wrong and it is destabilizing your software by Simon Timms
• You're (probably still) using HttpClient wrong and it is destabilizing your software by Josef Ottosson
• Why to Use IHttpClientFactory to Make HTTP Requests in .NET 5.0 or .NET Core by Rami Chalhoub

Last but not least, the official Microsoft documentation on how to use the HttpClient is a terrific resource: Make HTTP requests using IHttpClientFactory in ASP.NET Core.

The Issues

SocketException

The HttpClient implements IDisposable and when you encounter an IDisposable, you probably dispose of the resource as soon as possible like so:

using (var httpClient = new HttpClient())
{
  var result = await httpClient.GetAsync(...);
}

However, when the HttpClient is disposed of, a socket is tied up for some time. There are only a finite number of sockets, and under high load this can lead to SocketExceptions.

DNS

Despite the fact that HttpClient implements IDisposable, it's actually thread-safe and meant to be a reused, long-lived object. A well-intentioned solution is to use it as a singleton:

public static class HttpClientAccessor
{
  public static HttpClient HttpClient = new HttpClient();
}

var result = await HttpClientAccessor.HttpClient.GetAsync(...);

However this solution has its own issue: the HttpClient won't pick up DNS changes.

Below I show you how to tackle these issues on different versions of Sitecore.

Sitecore 8.1 and earlier

You're using a very old version of Sitecore. Mainstream support has ended and extended support has ended or is ending soon. Whatever patterns are being used in your solution probably aren't causing issues, so just keep using them. When in doubt just use System.Net.WebRequest; it's not sexy, but neither is the legacy code you're maintaining.

If you have implemented a dependency injection framework you can follow the suggestions below for Sitecore 8.2 to Sitecore 9.0 Update-2.

Sitecore 8.2 to Sitecore 9.0 Update-2

Sitecore 8.2 brought first-class dependency injection support into the platform, which you can take advantage of to use HttpClient safely.

Imagine that you have the following unsafe code in your codebase:

public class StrangeloveService
{
  public async Task<Bomb> GetBombAsync()
  {
    using (var httpClient = new HttpClient())
    {
      httpClient.BaseAddress = new Uri("https://lovethebomb.localhost/");
      var response = await httpClient.GetAsync(...);
      ...
    }
  }
}

Disposing of HttpClient on line 5 each time GetBombAsync is called can potentially lead to socket exhaustion under heavy load.

To address this, create an interface to access to your HttpClient:

public interface IStrangeloveHttpClientAccessor : IDisposable
{
  HttpClient HttpClient { get; }
}

And create an implementation of the interface:

public class StrangeloveHttpClientAccessor : IStrangeloveHttpClientAccessor
{
  public HttpClient HttpClient { get; }
  
  public StrangeloveHttpClientAccessor()
  {
    var address = new Uri("https://lovethebomb.localhost/");
    
    // Ensure DNS changes are picked up
    var servicePoint = ServicePointManager.FindServicePoint(address);
    servicePoint.ConnectionLeaseTimeout = 60 * 1000;
    
    HttpClient = new HttpClient
    {
      BaseAddress = address
    };
    
    // Do additional HttpClient configuration here such as default headers
  }
  
  public void Dispose()
  {
    HttpClient?.Dispose();
  }
}

Lines 10 and 11 are important as they will ensure that DNS changes are picked up for your HttpClient. In this example I set the connection lease timeout to 60 seconds, but you can increase or decrease based on your requirements.

Refactor the StrangeloveService to use the HttpClient accessor:

public class StrangeloveService
{
  private readonly IStrangeloveHttpClientAccessor _httpClientAccessor;

  public StrangeloveService(IStrangeloveHttpClientAccessor httpClientAccessor)
  {
    _httpClientAccessor = httpClientAccessor ?? throw new ArgumentNullException(nameof(httpClientAccessor));
  }

  public async Task<Bomb> GetBombAsync()
  {
    var httpClient = _httpClientAccessor.HttpClient;
    var response = await httpClient.GetAsync(...);
    ... 
  }
}

And register everything as singletons with the built-in dependency injection container:

public class StrangeloveServicesConfigurator : IServicesConfigurator
{
  public void Configure(IServiceCollection serviceCollection)
  {
    serviceCollection.AddSingleton<IStrangeloveHttpClientAccessor, StrangeloveHttpClientAccessor>();
    serviceCollection.AddSingleton<StrangeloveService>(); 
  }
}

If you're not familiar with registering services with Sitecore's dependency injection container or how to use IServicesConfigurator, see Sitecore's documentation on dependency injection.

Since the HttpClient accessor service is registered with the container as a singleton, there will be no SocketExceptions due to socket exhaustion, because only one HttpClient will be created. Additionally, DNS changes will be picked up regularly since we set the ConnectionLeaseTimeout on the ServicePointManager for the base address of the HttpClient.

With this pattern, you will need to create additional HttpClient accessors per HttpClient configuration (e.g., for different base addresses), which will likely mean an HttpClient accessor per API/service you use. That's okay, you won't create enough HttpClients this way to run into socket exhaustion.

Sitecore 9.1 and beyond

Sitecore 9.1 bumped the Microsoft.Extensions.DependencyInjection dependency up to 2.1.1, which paved the way to use Microsoft's solution to these HttpClient issues: the HttpClientFactory. Microsoft has a NuGet package called Microsoft.Extensions.Http with an HttpClientFactory that handles creation of HttpClients in a way that avoids both the socket exhaustion and DNS issues described above.

Use of the HttpClientFactory begins with installing the Microsoft.Extensions.Http NuGet package in your solution, but depending on the version of Sitecore you're working with, be careful what version of the NuGet package you install:

• Sitecore 9.1.* - 2.1.1
• Sitecore 9.2 - 2.1.1
• Sitecore 9.3 - 2.1.1
• Sitecore 10.0.* - 2.1.1
• Sitecore 10.1.* - 3.1.5
• Sitecore 10.2 - 3.1.14
• Sitecore 10.3 - 6.0.0

⚠️WARNING⚠️: Do not deploy a version of Microsoft.Extensions.Http or any of the Microsoft.Extensions.* packages to your Sitecore solution outside of the versions listed above. When in doubt, check the version of the Microsoft.Extensions.DependencyInjection assembly in your Sitecore install package and install the same version of Microsoft.Extensions.Http. Installing higher versions of these packages will cause issues, and is not supported by Sitecore.

Using the same the problematic service from above as an example:

public class StrangeloveService
{
  public async Task<Bomb> GetBombAsync()
  {
    using (var httpClient = new HttpClient())
    {
      httpClient.BaseAddress = new Uri("https://lovethebomb.localhost/");
      var response = await httpClient.GetAsync(...);
      ...
    }
  }
}

Regular creation and disposal of HttpClient will cause issues. Use the AddHttpClient extension method for IServiceCollection from the Microsoft.Extensions.Http NuGet package to register a named HttpClient for the StrangeloveService like so:

public class StrangeloveServicesConfigurator : IServicesConfigurator
{
  public void Configure(IServiceCollection serviceCollection)
  {
    serviceCollection.AddHttpClient(nameof(StrangeloveService), httpClient =>
    {
      httpClient.BaseAddress = new Uri("https://lovethebomb.localhost/");
    });
    serviceCollection.AddSingleton<StrangeloveService>();
  }
}

Again, if you're not familiar with registering services with Sitecore's dependency injection container or how to use IServicesConfigurator, see Sitecore's documentation on dependency injection.

The AddHttpClient extension method will register all of the HttpClientFactory machinery with Sitecore's dependency injection container if it hasn't been registered already.

The first parameter to AddHttpClient registers the name of the HttpClient with the factory--in this case we're using the name of the StrangeloveService type because we're configuring an HttpClient specifically for that class. The second parameter is an Action<HttpClient> that allows you to configure the HttpClient with things like BaseAddress, default headers, etc.

Update the StrangeloveService to depend on IHttpClientFactory and use the factory to create the client using the name in the StrangeloveServicesConfigurator above.

public class StrangeloveService
{
  private readonly IHttpClientFactory _httpClientFactory;

  public StrangeloveService(IHttpClientFactory httpClientFactory)
  {
    _httpClientFactory = httpClientFactory ?? throw new ArgumentNullException(nameof(httpClientFactory));
  }

  public async Task<Bomb> GetBombAsync()
  {
    var httpClient = _httpClientFactory.CreateClient(nameof(StrangeloveService));
    var response = await httpClient.GetAsync(...);
    ... 
  }
}

Behind the scenes, the HttpClientFactory will work its magic to reuse or recreate the HttpClient for StrangeloveService as needed to avoid socket exhaustion so you no longer need to worry about it. Never dispose of clients created by the HttpClientFactory--disposal and creation of HttpClient instances is the HttpClientFactory's responsibility.

Use this same pattern for each service/API that your application calls. Create named instances and configure them using the AddHttpClient extension method.

🚨DANGER🚨: The examples above are all completely async. Most Sitecore code is synchronous, meaning that you have to make your HttpClient calls synchronous, maybe by using httpClient.GetAsync(...).Result as shown below. DO NOT do this with the HttpClientFactory as you will introduce a deadlock into your code that can only be fixed with an application restart. Google around for best practices to run async methods synchronously and follow that guidance; never use code like below.

public class StrangeloveService
{
  private readonly IHttpClientFactory _httpClientFactory;

  public StrangeloveService(IHttpClientFactory httpClientFactory)
  {
    _httpClientFactory = httpClientFactory ?? throw new ArgumentNullException(nameof(httpClientFactory));
  }

  public Bomb GetBomb()
  {
    var httpClient = _httpClientFactory.CreateClient(nameof(StrangeloveService));
    var response = httpClient.GetAsync(...).Result;
    ... 
  }
}

ASP.NET Core Rendering SDK

The HttpClientFactory is a first-class citizen in the ASP.NET Core Rendering SDK and is ready for you to use without any extra configuration or NuGet packages to install.

Once again referencing the problematic code from before:

public class StrangeloveService
{
  public async Task<Bomb> GetBombAsync()
  {
    using (var httpClient = new HttpClient())
    {
      httpClient.BaseAddress = new Uri("https://lovethebomb.localhost/");
      var response = await httpClient.GetAsync(...);
      ...
    }
  }
}

In Startup.cs of your Rendering Host, register the StrangeloveService with the AddHttpClient extension method:

public void ConfigureServices(IServiceCollection services)
{
  ...
  services.AddHttpClient<StrangeloveService>(httpClient =>
  {
    httpClient.BaseAddress = new Uri("https://lovethebomb.localhost/");
  });
  ...
}

Notice that you pass StrangeloveService as the typed parameter to the extension method; this registers the StrangeloveService with the container as a transient service. You do not need to do services.AddTransient<StrangeloveService> after this call; it's handled for you. It will also register a named HttpClient specifically for the StrangeloveService.

The parameter to this extension method is an Action<HttpClient> that allows you to configure the HttpClient with things like BaseAddress, default headers, etc.

Registration is a lot simpler in the ASP.NET Core Rendering SDK, and so is use of the HttpClient. Rewrite the StrangeloveService like so:

public class StrangeloveService
{
  private readonly HttpClient _httpClient;
  
  public StrangeloveService(HttpClient httpClient)
  {
    _httpClient = httpClient ?? throw new ArgumentNullException(nameof(httpClient));
  }
  
  public async Task<Bomb> GetBombAsync()
  {
    var response = await httpClient.GetAsync(...);
    ...
  }
}

In the ASP.NET Core Rendering SDK you can inject HttpClient into your services directly! This is safe and possible because the AddHttpClient extension method registered StrangeloveService as a transient service--each time an instance of StrangeloveService is requested of the container, an existing or new instance of HttpClient is created specifically for StrangeloveService by the HttpClientFactory under the hood.

This code is definitely simpler, but keep reading as there's a gotcha.

Captive Dependencies

I addressed captive dependencies in a blog post a few years ago on the dangers of using scoped services in Sitecore. Microsoft also has a nice write up on the issue. In short:

1. You can inject singleton services into any service lifetime: singleton, scoped, or transient.
2. You can inject scoped services into scoped or transient services.
3. You can only inject transient services into transient services.

I call this out because if you use the method described above for the ASP.NET Core Rendering SDK, your services that depend on HttpClient (e.g., StrangeloveService) are registered as a transient service. This means that when you use this method, it is not safe to inject StrangeloveService into a scoped service or singleton service. If you do, best case scenario DNS changes will not be picked up; worst case scenario you'll get ObjectDisposedExceptions that are tough to track down.

The good news is that Microsoft has introduced a scope validation feature that can detect these some of issues at runtime. It is enabled by default in .NET 6 and .NET 7, and may be enabled by default in earlier versions though I have not validated myself. However, this functionality is quite limited (e.g., it only validates you don't inject scoped services into singletons), so even with this validation you may still run into ObjectDisposedExceptions under certain circumstances. If you're interested in using this feature in your Sitecore solutions, check out my blog post on the subject.

It's possible to use the ASP.NET Core Rendering SDK pattern in Sitecore 9.1 and beyond, but I do not recommend it. Services instantiated by Sitecore's Configuration Factory are singletons by default, which ensures that you're going to run into ObjectDisposedExceptions or DNS issues at some point if you use that method.

If you want to be absolutely sure you won't run into these issues, refer to the pattern I showed for Sitecore 9.1 and beyond. That pattern is always safe, even in the ASP.NET Core Rendering SDK, because all services are registered as singletons.