Test different configurations

Last Update: 4/7/2017

| Team Services |

Your users will probably install or run your app on a wide variety of configurations, such as different operating systems, web browsers, and other variations. You will want to run at least some of your tests in environments that have those different configurations.

Use your test plan to decide which tests you want to run on which configurations. You have to make sure that when you run your tests that you have set up your environments for the configurations that you need.

You might draw up a schematic matrix of the combinations that you want to test:

Test matrix of browser and operating system configurations

Then you can:

This feature is available only in Visual Studio Team Services.

Create configurations and variables

A test configuration is a combination of configuration variable values. Your configuration variables could be, for example, operating system, browser, CPU type, database. A configuration might be "Windows 8 + 32-bit CPU" or "Windows 10 + 64-bit CPU."

You must create the configuration variables first. Then combine multiple variable values to create a configuration.

  1. Open the Configurations tab in the Test hub, choose the new icon, and select New configuration variable.

    Adding a new configuration variable

  2. Type a name for the variable, such as Browser, and type a value. Add as many values as you wish to the configuration variable, and then save it.

    Setting the values for a new configuration variable

  3. Repeat the steps to create any other configuration variables you need. For example, create a configuration variable named Operating system with the names of each operating system on which you want to test.

    Setting the values for an Operating Systems configuration variable

  4. Choose the new icon and select New test configuration.

    Adding a new test configuration

  5. Type a name for the test configuration and add the configuration variables you created. Choose a value for each variable for this configuration.

    Adding variables to the new test configuration

    Ensure Assign to new test plans is checked to make this the default configuration for all the new test plans you create.

  6. Save your new test configuration.

Assign configurations to test plans and suites

You can assign configurations to a test plan, a test suite, or an individual test case. Configurations assigned to a test plan or test suite apply to all tests or suites within it.

  1. To assign a configuration to a test plan, open the shortcut menu for the plan and choose Assign configuration to test plan.

    Assigning a configuration to a test plan

  2. To assign a configuration to a test suite, open the shortcut menu for the suite and choose Assign configuration to test suite.

    Assigning a configuration to a test suite

    If you add multiple configurations to a test plan or suite, the tests cases are repeated in the plan or suite with the each of the configurations you have assigned.

    Multiple configuration assignment to a test suite

  3. If necessary, override the default configuration assigned to a test case and assign the configuration you need. Select one or more test cases, open the shortcut menu, and choose Assign configurations.

    Overriding the default configuration assigned to a test case

  4. Search for and select the configurations to assign to these test case(s).

    Setting the required configurations for a test case

Run tests with each configuration

  1. Set up a testing platform for a particular configuration, such as testing the app using Google Chrome on Windows 10.

  2. Select and run a test that has this configuration assigned.

    Running the test with the configuration assigned

    As you run the test, a reminder of the required configuration in shown in the status bar of the Test Runner window.

    The required configuration appears in the Test Runner window

Track test results for each configuration

  1. Open the Charts tab for your test plan or test suite, choose New, and select New test result chart.

    Creating a new test result chart

  2. Choose the type of chart you require, select Configuration in the Group by list, and choose OK.

    Choosing the type of chart you require

    A chart is created that can help you track your tests based on configurations. You can pin this chart to your dashboard.

    The configuration test result chart

Q&A

Q: I have a test case that appears in several test plans and test suites. Do I have to set the configurations for each of these test points?

A: Yes. The same test case can have different configuration settings in different test suites and test plans.

See also

Help and support

Submit bugs through Connect, make suggestions on Uservoice, and send quick thoughts using the Send-a-Smile icon link in the Visual Studio, Team Services, or TFS title bar. We look forward to your feedback.