Get started with Selenium testing in a CI pipeline

Last Update: 7/12/2017

Visual Studio 2017 | Visual Studio 2015 | TFS 2017 | TFS 2015

Performing user interface testing as part of the build process is a great way of detecting unexpected changes and need not be difficult. This topic describes using Selenium to test your website in a continuous integration build.

In this example, you'll learn how to:

For more information about Selenium browser automation, see:

Create the test project

As there is no template for Selenium testing, the easiest way to get started is to use the Unit Test template. This automatically adds the test framework references and enables you run and view the results from Visual Studio Test Explorer.

  1. In Visual Studio, open the File menu and choose New Project, then choose Test and select Unit Test Project. Alternatively, open the shortcut menu for the solution and choose Add then New Project and then Unit Test Project.

    For more details, see: Get started with unit testing.

  2. After the project is created, you must add the Selenium and browser driver references used by the browser to execute the tests. Open the shortcut menu for the Unit Test project and choose Manage NuGet Packages. Add the following packages to your project:

    • Selenium.WebDriver
    • Selenium.WebDriver.ChromeDriver
    • Selenium.WebDriver.IEDriver
    • Selenium.Firefox.WebDriver
    • Selenium.WebDriver.PhantomJS.Xplatform

    Adding the browser driver packages to your solution

  3. The Unit Test project creates a default class named UnitTest1.cs. To author a Selenium Test, replace the contents of the file with the following code.

      namespace Partsunlimited.UITests
      {
        using Microsoft.VisualStudio.TestTools.UnitTesting;
        using OpenQA.Selenium;
        using OpenQA.Selenium.Chrome;
        using OpenQA.Selenium.Firefox;
        using OpenQA.Selenium.IE;
        using OpenQA.Selenium.Remote;
        using OpenQA.Selenium.PhantomJS;
        using System;
    
        [TestClass]
        public class ChucksClass1
        {
          private string baseURL = "http://your-website.azurewebsites.net/";
          private RemoteWebDriver driver;
          private string browser;
          public TestContext TestContext { get; set; }
    
          [TestMethod]
          [TestCategory("Selenium")]
          [Priority(1)]
          [Owner("FireFox")]
    
          public void TireSearch_Any()
          {
            driver = new FirefoxDriver();
            driver.Manage().Window.Maximize();
            driver.Manage().Timeouts().ImplicitlyWait(TimeSpan.FromSeconds(30));
            driver.Navigate().GoToUrl(this.baseURL);
            driver.FindElementById("search - box").Clear();
            driver.FindElementById("search - box").SendKeys("tire");
            //do other Selenium things here!
          }
    
          [TestCleanup()]
          public void MyTestCleanup()
          {
            driver.Quit();
          }
    
          [TestInitialize()]
          public void MyTestInitialize()
          {
          }
        }
      }
    

    Replacing the code in UnitTest1.cs

  4. Run the Selenium test locally using Test Explorer.

    Running the tests in Visual Studio Test Explorer

Include the test in a CI build

To include the Selenium test as part of a build, the source code must be in version control.

Checking the code into Visual Studio Team Services

  1. In your Visual Studio Team Services account where you checked in the test code, open the Build & Release hub and select the Builds tab.

  2. Create a new build definition using the .NET Desktop build template.

  3. In the new build definition, select the Default agent queue in which you have installed an agent. If you have not installed an agent in the Default queue, choose the manage queues link and do that now. For information, see Deploy a Windows build agent. You might decide to create an Azure VM to install your agent, or use a deployment group and install an agent there.

    Selenium tests will generally be run interactively, which would fail on the Hosted build controller.

  4. In the Get sources step, ensure that the repository and branch where you checked in your code is selected.

  5. Select the Triggers tab and turn on the Continuous integration... trigger so that your solution builds after each check-in using continuous integration.

  6. Delete the Test Assemblies (Visual Studio Test) task step from the build definition, then add a Visual Studio Test Agent Deployment, Windows Machine File Copy, and Run Functional Tests task from the Test and Deploy tabs of the task catalog. Drag and drop then in that order immediately after the Publish symbols path task step.

    Tasks in the build definition

  7. Configure the tasks as shown here:

    NuGet Package: NuGet - Install and update NuGet package dependencies.

    • Path to solution, packages.config, or project.json: Select your app solution (.sln) file.

    • Feeds to use: Feeds I select here

    • Use packages from NuGet.org: Checked

    Visual Studio Build Build: Visual Studio Build - Build with MSBuild and set the Visual Studio version property.

    • Solution: Select your app solution (.sln) file.

    • Platform: $(BuildPlatform)

    • Configuration: $(BuildConfiguration)

    • Visual Studio Version: Select the version used to create your app.

    Index Sources & Publish Symbols Test: Index Sources & Publish Symbols - Index the source code and optionally publish symbols to a SymStore file share.

    • Search pattern: **\bin\**\*.pdb

    Visual Studio Test Agent Deployment Test: Visual Studio Test Agent Deployment - Deploy and configure the test agent to run tests on a set of machines.

    • Machines: Comma-delimited list of machine names, or a variable containing the list.

    • Admin Login: Username for target server or a variable containing it.

    • Admin Password: Password for target server or a variable containing it.

    • Protocol: HTTP

    • Select Machines By: Machine Names

    • Agent Configuration - Username: Agent username or a variable containing it.

    • Agent Configuration - Password: Agent password or a variable containing it.

    • Agent Configuration - Interactive Process: Checked

    Windows Machine File Copy Deploy: Windows Machine File Copy - Copy files to remote machines.

    • Source: $(Build.Repository.LocalPath)

    • Machines: Comma-delimited list of machine names, or a variable containing the list.

    • Admin Login: Username for target server or a variable containing it.

    • Password: Password for target server or a variable containing it.

    • Destination Folder: C:\Deploy or another folder on the target server.

    Run Functional Tests Test: Run Functional Tests - Run Coded UI tests, Selenium tests, and functional tests on a set of machines using the test agent.

    • Machines: Comma-delimited list of machine names, or a variable containing the list.

    • Test Drop Location: C:\Deploy or the folder where you copied the files if different.

    • Execution Options - Test Selection: Test Assembly

    • Execution Options - Test Assembly: **\*Test*.dll

    Copy Files Test: Copy Files - Copy files from a source folder to a target folder using match patterns.

    • Source Folder: $(build.sourcesdirectory)

    • Contents: **\bin\$(BuildConfiguration)\**

    • Target Folder: $(build.artifactstagingdirectory)

    Publish Build Artifacts Test: Publish Build Artifacts - Publish Build artifacts to the server or a file share.

    • Path to Publish: Select your Azure subscription.

    • Artifact Name: drop

    • Artifact Type: Server

    It's generally advisable to use custom variables for parameter values, especially where the same value is used in the parameters of more than one task. You can also secure and hide values by using custom variables. See Build Variables.

  8. Save the build definition and queue a new build.

View the test results

  1. To view the test results from a build, open the build summary from the Builds tab.

    Selecting a build result

    The build summary includes a snapshot of the test results. There is also a Tests results page that highlights the build-on-build changes, including errors, stack traces, and the ability to easily create a bug that contains this information.

    The build summary and test results

Next steps

Also see

For more information about Selenium browser automation, see:

For information about deploying PhantomJS.exe as part of your test, see this blog post.

Help and support

Report any problems on Developer Community, make suggestions on UserVoice, get advice on Stack Overflow, and get support via our Support page. We look forward to your feedback.