d

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore.

15 St Margarets, NY 10033
(+381) 11 123 4567
ouroffice@aware.com

 

KMF

In-memory Automated UI Testing ASP.NET Core

Introduction

In this article, we look at how to run in-memory automated UI tests for an ASP.NET Core web app using Playwright and NUnit. The article provides demo code and solutions to issues found along the way.

Automated UI Testing

Automated testing of any web application is essential to ensure it functions correctly. ¬†On the top of the ‚Äútesting pyramid‚ÄĚ proudly sits UI testing or end-to-end testing, above integration and unit testing. ¬†Automated UI testing involves launching and controlling a browser to drive through a set of interactions that a user would perform. ¬†Assertions are made to see if the web app and browser behave as expected. ¬†Browsers are controlled by testing tools such as Selenium, ¬†Puppeteer, or the new kid on the block, Playwright.

Normally, tests are run against a web application deployed to a test environment that has been configured with the same infrastructure as the production environment.  However, in this post, we will be looking at running our UI tests against an in-memory instance of a web application.  By running the test in this way, we can:

  • Develop tests faster.
  • Reconfigure the app with test configuration.
  • Catch some errors earlier in the dev/test process.
  • Still run the tests against a deployed environment later.

Before we get started, we need to take a small step back.

Integration Testing With WebApplicationFactory

Microsoft provides a neat way to run integration tests using an in-memory webserver called TestServer.

Using the WebApplicationFactory class will provide you with a running TestServer.  It requires a type in the entry point assembly of the application. Typically the Startup or Program classes can be used.  The WebApplicationFactory class also provides a virtual ConfigureWebHost() method, which allows for the web host to be manipulated.  This is powerful and means services can be removed, added, or swapped.  For example, a SQL Server database can be removed and replaced with an in-memory database.

A unit test framework such as xUnit or NUnit can be used to write and run the tests targeting the app hosted in the in-memory TestServer. ¬†When tests run they use an HttpClient connection given by the TestServer. ¬†The HttpClient is a C# class and is not a browser, which means it’s a bit limited and inconvenient to use.

Using the TestServer approach, we have an in-memory web server at our disposal.  Not only that, it provides a way to override the way the web host is built so that we can customize our application for our testing needs.  That sounds great! Unfortunately, when using a browser automation tool, such as Selenium or Playwright, the TestServer approach does not quite work.

Modifying Test Server to Work With Selenium and Playwright

After reading some really useful articles by Ben Day, Bertrand Thomas, and an oldie from Scott Hanselman (thanks guys!), I’ve derived a class from WebApplicationFactory which works with browser testing tools including Selenium and Playwright. In the demo project, this is the AutomatedTestServerFactory class.

One of the main issues when using a TestServer with a browser testing tool is that ConfigureWebHost() is not called. ¬†This means the app cannot be changed at startup to use a test configuration. ¬†In the ‚Äútraditional‚ÄĚ way of using TestServer, we are given an HttpClient¬†instance for our request/response interaction with the server. ¬†The HttpClient¬†instance is created using the CreateClient() method in WebApplicationFactory, which in turn runs the ConfigureWebHost() method. ¬†So, the really useful re-configuration we need is only done when an HttpClient¬†is created! Browser testing tools effectively replace the client and are controlled independently of the TestServer.

To get around this issue AutomatedTestServerFactory  sets the configuration when the web host is built.  The work is mainly done in two methods CreateServer() and CreateWebHostBuilder(), which are called by the constructor.

CreateWebHostBuilder() is called first and sets the web host configuration, which is passed through as a constructor parameter.  Also optionally set is the environment, otherwise, the default environment is production, which may not be ideal.

Test Server optimization for Selenium and Playwright

CreateServer() builds and starts the web application and begins listening for requests. This method returns a null TestServer.  I didn’t see any impact of this and it avoids a new and different instance of the web app from being created.

Selenium ad Playwright compatible WebApplicationFactory

With these changes in place, we have a WebApplicationFactory that creates an in-memory TestServer hosting our re-configured web application and is suitable for use with Selenium or Playwright.   Super!

Using Selenium or Playwright

When developing this project, I started using Selenium as the browser testing tool, however, I noticed in October 2021, that Microsoft updated the integration test article (see above) to recommend Playwright instead of Selenium.  So I took a look, I liked it, and I have used Playwright for the remainder of the project. The Playwright is a relatively new end-to-end testing tool developed and maintained by Microsoft.

The reasons I liked Playwright include its simplification of browser driver management, that it works with Chromium, Firefox, and Webkit browsers, and the ease of use of the API compared to Selenium’s.  Out of interest, I found migrating tests from Selenium to Playwright to be straightforward too!

The decision to use Playwright also led to selecting NUnit as the test runner. ¬†Typically, I use xUnit for testing, but it doesn’t support running Playwright tests in parallel.

Configuring Playwright on local PC

To get Playwright to work, there are two steps required using Package Manager Console:

  1. Install Playwright CLI as a global tool.
    • Run the command: install -g Microsoft.Playwright.CLI
  2. Install browsers.
    • Navigate to the path of the test project and run the command: playwright install

Code Time!

In order to demonstrate this work, I’ve created a demo application available on GitHub.  The codebase has three parts as described below.

Employee System

This app is a very simple ASP.NET Core MVC application using EntityFrameworkCore with SqlClient. The only functionality is a homepage and a page returning a list of employees.  This is enough for our purposes though!  It has a migration script, so remember to run update-database before launching it.

This is what the Employee List page looks like:

Employee list page

Test code

The test code is comprised of two libraries, AutomatedTesting, and Employees.UITests.  The AutomatedTesting library contains the custom AutomatedTestServerFactory class to enable in-memory testing with Selenium or Playwright. 

Employees.UITests is an NUnit test library.  It contains a base class, BaseTestClass, which is responsible for creating the webserver and passing in the required configuration, in this case changing the database to in-memory and adding test data.  The base class is decorated with the attribute [Parallelizable(ParallelScope.Children)] to allow child tests to run in parallel.

Running test project

The EmployeeListTests class inherits from BaseTestClass and focusses on the tests, so the code is quite clean. There are three tests, each one is the same, but using different browsers.  Running the test project will cause all three tests to run in parallel.  Finally, I’ve used FluentAssertions (as I like it!) for evaluating the outcomes of tests.

In the excerpt below, the Employee menu option is clicked and assertions are made on the output.

Employee menu assertions on the output

I have started to pull out some common tasks in TestHelper, such as taking a screenshot and attaching it to the test report. After running the tests, you can see the results in the test log by right-clicking on a test and opening the log.

Test results in the test log

In Visual Studio 2019, the results include both the log messages and the screenshots. 

Test results in Visual Studio 2019

DevOps

I have included a yaml build pipeline for Azure DevOps.  This configuration deals with the Playwright requirements builds the code and executes the tests in a cloud-hosted Windows server.  It can target a self-hosted Windows target too.

Resolved Issues

Developing this project was not without its issues.  Apart from getting the AutomatedTestServerFactory to work, here are the issues I found and overcame.

Issues Found on Developer PC

Issue 1

The web application launches, but is missing styling, JavaScript, and images. This was because the test server would launch in the bin directory of the test project and not have a reference to the wwwroot folder. 

Issues found on developer PC - 1

Solution: This was resolved by adding a post-build target to the Employees.UITests project file. The target copies the wwwroot folder recursively from the Employees.UI project to the Employees.UITestsbin folder.

Solution to issues found on developer’s PC - 1

Microsoft has an example of how to copy files recursively during the after build. 

Issue 2

Microsoft.Playwright.PlaywrightException : Executable doesn’t exist

Solution: One or more of the browsers is missing.  Run playwright install command in the Package Manager Console, but significantly make sure this is done in the directory of the test project.

Issues Found in Azure DevOps

Issue 3

Install step of Playwright Tool causes an error when re-run.  The following two lines in the log file indicate the issue.

Tool 'microsoft.playwright.cli' is already installed.
Error: The process 'C:Program Filesdotnetdotnet.exe' failed with exit code 1

Issues found in Azure DevOps

Solution: Instead of calling dotnet install, run the dotnet update command as this removes and clean installs the tool.

Issue 4

Error: Cannot find a tool in the manifest file that has a command named ‘playwright’.

Solution: Make sure the playwright install command runs in the directory of the test project.

Issue 5

Cloud-hosted Chrome fails with exception: 

Microsoft.Playwright.PlaywrightException : net::ERR_CERT_AUTHORITY_INVALID at https://localhost:5001/

Solution: Get a new browser instance to get around this issue using the code below.  This effectively runs the tests in incognito mode.

await browser.NewContextAsync(new BrowserNewContextOptions { IgnoreHTTPSErrors = true });

Issue 6

Firefox does not navigate to the test site.  Instead, a warning is displayed in the browser

‚ÄúWarning: Potential Security Risk Ahead‚ÄĚ.

Solution: Firefox prevents navigation to another port on localhost.  The warning requires manual intervention, but the settings can be overridden so that the automated test can run.  To do so, set the following capabilities (these settings are briefly covered here):

caps.Add("security.insecure_field_warning.contextual.enabled", false);

caps.Add("security.certerrors.permanentOverride", false);

caps.Add("network.stricttransportsecurity.preloadlist", false);

caps.Add("security.enterprise_roots.enabled", true);

Other Points

  1. Firefox not waiting for AJAX request to complete.

When waiting for a page to load that fires off an AJAX request, for example fetching data on page load, Firefox does not seem to wait.  To resolve it, add await WaitForLoadStateAsync() after the command to load the page.

  1. Azure DevOps only includes NUnit log on test failure.

NUnit provides the convenient TestContext.Out to log messages during testing.  These will always appear using the test runner in Visual Studio 2019, but these messages are only present in Azure DevOps when a test fails.

Conclusion

In this article, we have explored the code, configuration, and gotchas of running automated UI tests of an ASP.NET Core web app using Microsoft’s TestServer.  We have been able to perform UI testing using Playwright with multiple browsers running in parallel using NUnit.

We have seen a number of issues, mainly arising in Azure DevOps, and how to overcome them. In-memory automated UI testing could be great for smaller systems and to help develop test code. It’s not an approach for all cases but certainly has its merits. Finally, if you‚Äôve not used Playwright for automated testing, perhaps it is a good opportunity to try it out!

Credit: Source link

Previous Next
Close
Test Caption
Test Description goes like this