https://andrewlock.net/exploring-dotnet-6-part-6-supporting-integration-tests-with-webapplicationfactory-in-dotnet-6/
This is the sixth post in the series: Exploring .NET 6.
- Part 1 - Looking inside ConfigurationManager in .NET 6
- Part 2 - Comparing WebApplicationBuilder to the Generic Host
- Part 3 - Exploring the code behind WebApplicationBuilder
- Part 4 - Building a middleware pipeline with WebApplication
- Part 5 - Supporting EF Core migrations with WebApplicationBuilder
- Part 6 - Supporting integration tests with WebApplicationFactory in .NET 6 (this post)
- Part 7 - Analyzers for ASP.NET Core in .NET 6
- Part 8 - Improving logging performance with source generators
- Part 9 - Source generator updates: incremental generators
- Part 10 - New dependency injection features in .NET 6
- Part 11 - [CallerArgumentExpression] and throw helpers
- Part 12 - Upgrading a .NET 5 "Startup-based" app to .NET 6
In the previous post I described the workaround that was added in .NET 6 so that the EF Core tools, which previously relied on the existence of specific methods like CreateHostBuilder would continue to work with the new minimal hosting APIs.
In this post I look at a related change to ensure that integration testing with WebApplicationFactory works in .NET 6. WebApplicationFactory used the same HostFactoryResolver class as the EF Core tools, but it required a few more changes too, which I'll look at in this post.
WebApplicationFactory in ASP.NET Core 3.x/5
There are multiple ways to test an ASP.NET Core 3.x/5 app. One of the most thorough approaches is to write integration tests that run your whole application in-memory. This is surprisingly easy using the Microsoft.AspNetCore.Mvc.Testing package and WebApplicationFactory<T>.
For example, the following code, based on the docs, shows how you can use WebApplicationFactory to create an in-memory instance of your application, create an HttpClient for making requests, and send an in-memory HTTP request.
public class BasicTests : IClassFixture<WebApplicationFactory<Startup>>
{
private readonly WebApplicationFactory<Startup> _factory;
public BasicTests(WebApplicationFactory<Startup> factory)
{
_factory = factory;
}
[Fact]
public async Task Get_EndpointsReturnSuccessAndCorrectContentType()
{
// Arrange
var client = _factory.CreateClient();
// Act
var response = await client.GetAsync("/");
// Assert
response.EnsureSuccessStatusCode(); // Status Code 200-299
}
}
Behind the scenes, WebApplicationFactory<T> uses the same HostFactoryResolver I described in the previous post. The generic parameter TEntryPoint is conventionally set to Startup, but it just needs to be a type inside the entry assembly, so that it can find the CreateHostBuilder() method:
public class WebApplicationFactory<TEntryPoint> : IDisposable where TEntryPoint : class
{
protected virtual IHostBuilder CreateHostBuilder()
{
var hostBuilder = HostFactoryResolver.ResolveHostBuilderFactory<IHostBuilder>(typeof(TEntryPoint).Assembly)?.Invoke(Array.Empty<string>());
if (hostBuilder != null)
{
hostBuilder.UseEnvironment(Environments.Development);
}
return hostBuilder;
}
// ...
}
As I described in my previous post, HostFactoryResolver uses reflection to find the conventionally named methods CreateHostBuilder() or CreateWebHostBuilder() and and invoke them. However in .NET 6 the minimal hosting APIs and top level programs have done away with these conventions, initially breaking WebApplicationFactory (along with the EF Core tools).
Building an IHost in .NET 6
In the previous post I described the changes that were made to HostBuilder to support the HostFactoryResolver, used by both WebApplicationFactory and the EF Core tools. This was primarily achieved by adding additional DiagnosticSource events to HostBuilder. These provide a way for HostFactoryResolver to get access to the HostBuilder without needing the conventions of previous versions.
WebApplicationFactory benefits from this new mechanism too, but there were a few additional changes needed. The EF Core tools just needed to access the built IHost so that they could retrieve an IServiceProvider. The IServiceProvider is fixed once you build the IHost, so the "abort the application" approach shown in the above image worked just fine.
However, that doesn't work for the WebApplicationFactory. WebApplicationFactory needs be able to modify the HostBuilder in your application before Build() is called on it, but it also can't just stop your program when HostBuilder.Build() is called.
In .NET 6 you can (and do) write all sorts of code between the call to WebApplicationBuilder.Build() and WebApplication.Run(). You can't modify the IServiceCollection, but you can register your endpoints and middleware between these calls:
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddRazorPages();
var app = builder.Build(); // calls HostBuilder.Build()
app.UseStaticFiles();
app.MapGet("/", () => "Hello World!");
app.MapRazorPages();
app.Run(); // calls Host.StartAsync()
This makes things a bit trickier for WebApplicationFactory, as it needs to run all the code in Program.cs, right up until the call to app.Run(), so it can't rely on the DiagnosticSource events added to HostBuilder alone. In the rest of this post, we'll look at how WebApplicationFactory achieves that.
WebApplicationFactory in .NET 6
On the surface, the way you use WebApplicationFactory hasn't changed in .NET 6. The exact same test code I showed earlier will work in .NET 6, even if you're using the new minimal hosting APIs with WebApplication and WebApplicationBuilder.
One slight annoyance is that there's no well known
Startupclass to use as a "marker" for theTgeneric parameter inWebApplicationFactory<T>. In practice, this is probably only an issue for demo-ware, as you can use any old class in your web app as a marker, but it's something to be aware of.
WebApplicationFactory provides multiple ways to customise your application in integration tests, but at it's core, it provides a way to run your application's Host instance in memory. One of the core methods in this process is EnsureServer(), which is partially shown below.
public class WebApplicationFactory<TEntryPoint> : IDisposable, IAsyncDisposable where TEntryPoint : class
{
private void EnsureServer()
{
// Ensure that we can find the .deps.json for the application
EnsureDepsFile();
// Attempt to create the application's HostBuilder using the conventional
// CreateHostBuilder method (used in ASP.NET Core 3.x/5) and HostFactoryResolver
var hostBuilder = CreateHostBuilder();
if (hostBuilder is not null)
{
// If we succeeded, apply customisation to the host builder (shown below)
ConfigureHostBuilder(hostBuilder);
return;
}
// Attempt to create the application's WebHostBuilder using the conventional
// CreateWebHostBuilder method (used in ASP.NET Core 2.x) and HostFactoryResolver
var builder = CreateWebHostBuilder();
if (builder is null)
{
// Failed to create the WebHostBuilder, so try the .NET 6 approach
// (shown in following section)
// ...
}
else
{
// succeeded in creating WebHostBuilder, so apply customisation and exit
SetContentRoot(builder);
_configuration(builder);
_server = CreateServer(builder);
}
}
private void ConfigureHostBuilder(IHostBuilder hostBuilder)
{
// Customise the web host
hostBuilder.ConfigureWebHost(webHostBuilder =>
{
SetContentRoot(webHostBuilder);
_configuration(webHostBuilder);
// Replace Kestrel with TestServer
webHostBuilder.UseTestServer();
});
// Create the IHost
_host = CreateHost(hostBuilder);
// Retrieve the TestServer instance
_server = (TestServer)_host.Services.GetRequiredService<IServer>();
}
}
EnsureServer() is responsible for populating the TestServer field _server using HostFactoryResolver. It first tries to create an IHostBuilder instance by looking for the Program.CreateHostBuilder() method, commonly used in ASP.NET Core 3.x/5. If that fails, it looks for the Program.CreateWebHostBuilder() method used in ASP.NET Core 2.x. And if that fails, it resorts to the .NET 6 approach, which I've extracted from the method above, and is shown below.
// Create a DeferredHostBuilder, which I'll discuss shortly
var deferredHostBuilder = new DeferredHostBuilder();
deferredHostBuilder.UseEnvironment(Environments.Development);
// Ensure the application name is set correctly. Without this, the application name
// would be set to the testhost (see https://github.com/dotnet/aspnetcore/pull/35101)
deferredHostBuilder.ConfigureHostConfiguration(config =>
{
config.AddInMemoryCollection(new Dictionary<string, string>
{
{ HostDefaults.ApplicationKey, typeof(TEntryPoint).Assembly.GetName()?.Name ?? string.Empty }
});
});
// This helper call does the hard work to determine if we can fallback to diagnostic source events to get the host instance
var factory = HostFactoryResolver.ResolveHostFactory(
typeof(TEntryPoint).Assembly,
stopApplication: false,
configureHostBuilder: deferredHostBuilder.ConfigureHostBuilder,
entrypointCompleted: deferredHostBuilder.EntryPointCompleted);
if (factory is not null)
{
// If we have a valid factory it means the specified entry point's assembly can potentially resolve the IHost
// so we set the factory on the DeferredHostBuilder so we can invoke it on the call to IHostBuilder.Build.
deferredHostBuilder.SetHostFactory(factory);
ConfigureHostBuilder(deferredHostBuilder);
return;
}
// Failed to resolve the .NET 6 entrypoint, so failed at this point
throw new InvalidOperationException();
This method uses a new type, DeferredHostBuilder, which we'll look into shortly, but the important section is the call to HostFactoryResolver.ResolveHostFactory(). This is the method that uses the DiagnosticSource events I discussed in my last post to customise the IHostBuilder and to access the IHost. Specifically, the call registers two callbacks:
deferredHostBuilder.ConfigureHostBuilder: called just before theIHostBuilderis built, and passed theIHostBuilderinstance.deferredHostBuilder.EntryPointCompleted: called if an exception occurs during the build process.
Importantly, the stopApplication argument is set to false; this ensures the application startup process continues uninterrupted.
Contrast that with the EF Core tools approach, in which
stopApplication=true. The EF Core tools don't want to run your application, they just need access to theIHost(andIServiceProvider), so they can halt after these are built.
The following diagram shows the interaction with WebApplicationFactory, the HostFactoryResolver, and DeferredHostBuilder, as well as other types we'll see on the way. Don't worry about understanding this fully for now, but I think it's helpful to view now as a signpost for where we're going!
You might wonder why we need a new type here, the DeferredHostBuilder. This is necessary because of the asynchronous way we have to wait for the "main" application to finish running. In the next section I'll look at this type in detail.
DeferredHostBuilder and waiting for the StartAsync signal
DeferredHostBuilder is YAHB (Yet Another IHostBuilder) that was introduced in .NET 6 (along with many others)! It's designed to "capture" configuration methods called on it (such as ConfigureServices() for example) and then "replay" them against the real application's IHostBuilder once it's available.
The "deferral" methods work by collecting the configuration methods as a multi-cast delegate, for example:
internal class DeferredHostBuilder : IHostBuilder
{
private Action<IHostBuilder> _configure;
public IHostBuilder ConfigureServices(Action<HostBuilderContext, IServiceCollection> configureDelegate)
{
_configure += b => b.ConfigureServices(configureDelegate);
return this;
}
// ...
}
These delegates are all applied to the IHostBuilder when the DiagnosticSource HostBuilding event is raised:
public void ConfigureHostBuilder(object hostBuilder)
{
_configure(((IHostBuilder)hostBuilder));
}
To get this all rolling, the WebApplicationFactory calls Build() on the DeferredHostBuilder. This method, as shown below, invokes the _hostFactory method returned by HostResolverFactory. Calling this method starts the process described in the previous post, in which the application is executed in a separate thread, the ConfigureHostBuilder() customisation is called using DiagnosticSource events, and the IHost instance is returned. That's a lot for a single line of code!
public IHost Build()
{
// Hosting configuration is being provided by args so that
// we can impact WebApplicationBuilder based applications.
var args = new List<string>();
// Transform the host configuration into command line arguments
foreach (var (key, value) in _hostConfiguration.AsEnumerable())
{
args.Add($"--{key}={value}");
}
// Execute the application in a spearate thread, and listen for DiagnosticSource events
var host = (IHost)_hostFactory!(args.ToArray());
// We can't return the host directly since we need to defer the call to StartAsync
return new DeferredHost(host, _hostStartTcs);
}
Remember that the application doesn't stop running in the separate Thread when we retrieve the IHost instance, because we need the rest of the code in Program.cs to execute. The DeferredHostBuilder saves the IHost into a new type, DefferedHost, and returns this from the Build() call.
_hostStartTcsis aTaskCompletionSourcethat is used to handle the edge case where the application running in the background exits due to an exception. It's an edge case, but without it, the test could hang indefinitely.
The DeferredHost is responsible for waiting for the application to properly start (not just for the IHost to be built). It needs to wait for you to configure all your endpoints, as well as run any additional startup code.
The DeferredHost achieves this by using the existing IHostApplicationLifetime events that are raised in a normal generic host application when the app is started. The following image (taken from a previous post analysing the startup process for the generic host) shows that the NotifyStarted() method is called on the IHostApplicationLifetime after the server has started.
The call to NotifyStarted() raises the ApplicationStarted event, which the DeferredHost uses to detect that the application is running, and it's safe to start the test. When the WebApplicationFactory calls Start() on the DeferredHost, the DeferredHost blocks until the ApplicationStarted event is raised.
public async Task StartAsync(CancellationToken cancellationToken = default)
{
// Wait on the existing host to start running and have this call wait on that. This avoids starting the actual host too early and
// leaves the application in charge of calling start.
using var reg = cancellationToken.UnsafeRegister(_ => _hostStartedTcs.TrySetCanceled(), null);
// REVIEW: This will deadlock if the application creates the host but never calls start. This is mitigated by the cancellationToken
// but it's rarely a valid token for Start
using var reg2 = _host.Services.GetRequiredService<IHostApplicationLifetime>().ApplicationStarted.UnsafeRegister(_ => _hostStartedTcs.TrySetResult(), null);
await _hostStartedTcs.Task.ConfigureAwait(false);
}
The StartAsync() method adds additional callbacks to the TaskCompletionSource I mentioned previously, and then awaits the task before returning. This will block the test code until one of three things happen:
- The web application throws an exception, which invokes the
EntryPointCompleted()callback onDeferredHostBuilderand cancels the task. - The
CancellationTokenpassed to theStartAsync()method is cancelled, which cancels the task - The application starts, invoking the
ApplicationStartedevent, completing the task.
As noted in the comments for this method, if you never call Start() or Run() in your web app (and don't throw an exception), then this will deadlock, but then you probably don't have a valid web app anyway, so it's not a big concern.
And that's it! Once Start() has been called, the WebApplicationFactory creates an HttpClient as in previous versions, and you can make in-memory calls as before. It's worth being aware that (in contrast to previous versions of ASP.NET Core) everything in Program.cs will be running in your tests. But apart from that, everything in your test code stays the same.
Summary
In this post I described the work that was done on WebApplicationFactory to support the new minimal host APIs that use WebApplication and WebApplicationBuilder. Changes were required because there are no longer "conventional" methods in Program.cs that can be called using reflection, and also customisation of your middleware and endpoints occurs inside Program.cs.
To work around this, WebApplicationFactory relies on the same DiagnosticSource events as the EF Core tools from my previous post to customise the IHostBuilder and retrieve the IHost. However, unlike the EF Core tools, the WebApplicationFactory does not stop the application after the IHost is built. Instead, it allows the app to continue to run, and listens for the IHostApplicationLifetime.ApplicationStarted event. This allows the WebApplicationFactory to block until all the code in Program.cs has run, and the application is ready to start handling requests.