Team Services | TFS 2017 | TFS 2015 Update 3 | Previous versions (XAML builds)
Run Coded UI tests, Selenium tests, and functional tests on a set of machines using the test agent. Use this task when you want to run tests on remote machines, and you cannot run tests on the build machine.
Demands and prerequisites
This task must be preceded by a Visual Studio Test Agent Deployment task.
A comma-separated list of machine FQDNs or IP addresses, optionally including the port number. Can be:
|Test Drop Location||
Required. Whether the tests are to be selected from test assemblies or from a test plan.
Required when Test Selection is set to Test Assembly. The test assemblies from which the tests should be executed.
|Test Filter criteria||
Optional when Test Selection is set to Test Assembly. A filter to specify the tests to execute within the test assembly files.
Required if Test Suite is not specified when Test Selection is set to Test Plan. Select a test plan already configured for this account.
Required if Test Plan is not specified when Test Selection is set to Test Plan. Select a test suite from the selected test plan.
Optional when Test Selection is set to Test Plan. Select a test configuration from the selected test plan.
|Run Settings File||
Optional. The path to a
|Override Test Run Parameters||
Optional. A string containing parameter overrides
for parameters defined in the
|Code Coverage Enabled||
When set, the task will collect code coverage information during the run and upload the results to the server.
|Distribute tests by number of machines||
When checked, distributes tests based on the number of machines, instead of distributing tests at the assembly level, irrespective of the container assemblies passed to the task.
|Test Run Title||
Optional. A name for this test run, used to identify it for reporting and in comparison with other test runs.
Optional. The build platform against which the test run should be reported.
Optional. The build configuration against which the test run should be reported.
Optional. A string that contains the filter(s) to report the configuration on which the test case was run.
|Application Under Test Machines||
A list of the machines on which the Application Under Test (AUT) is deployed, or on which a specific process such as W3WP.exe is running. Used to collect code coverage data from these machines. Use this in conjunction with the Code Coverage Enabled setting.
The list can be:
Typical scenarios include:
- Tests that require additional installations on the test machines, such as different browsers for Selenium tests
- Coded UI tests
- Tests that require a specific operating system configuration
- To execute a large number of unit tests more quickly by using multiple test machines
Use this task to:
- Run automated tests against on-premises standard environments
- Run automated tests against existing Azure environments
- Run automated tests against newly provisioned azure environments
You can run unit tests, integration tests, functional tests - in fact any test that you can execute using the Visual Studio test runner (vstest).
Using multiple machines in a Machine Group enables the task to run parallel distributed execution of tests. Parallelism is at the test assembly level, not at individual test level.
These scenarios are supported for:
TFS on-premises and Visual Studio Team Services
** Build agents**
- Hosted and on-premises agents.
- Using the XPlat agent is not supported for any Behavior Driver Test (BDT) tasks.
- The build agent must be able to communicate with all test machines. If the test machines are on-premises, the hosted build agent cannot be used.
- The build agent must have access to the Internet to download test agents. If this is not the case, the test agent must be manually downloaded and deployed to a network location that is accessible by the build agent, and a Visual Studio Test Agent Deployment task used with an appropriate path for the Test Agent Location parameter. Automatic checking for new test agent versions is not supported in this topology.
** CI/CD workflow**
- The BDT tasks are supported in both build and release management definitions.
** Machine group configuration**
- Only Windows machines are supported when using BDT tasks inside a Machine Group. Using Linux, iOS, or other platforms inside a Machine Group with BDT tasks is not supported.
- Installing any version or release of Visual Studio on any of the test machines is not supported.
- Installing an older version of the test agent on any of the test machines is not supported.
** Test machine topologies**
- Azure-based test machines are fully supported, both existing test machines and newly provisioned machines.
- Domain-joined test machines are supported.
- Workgroup-joined test machines must have HTTPS authentication enabled and configured during creation of the Machine Group.
- Test agent machines must have network access to the Team Foundation Server instance. Test machines isolated on the network are not supported.
** Usage Error Conditions**
- Running tests across different Machine Groups, and running builds (with any BDT tasks) in parallel against these Machine Groups is not supported.
- Cancelling an in-progress build or release with BDT tasks is not supported. If you do so, subsequent builds may not behave as expected.
- Cancelling an in-progress test run queued through BDT tasks is not supported.
- Configuring a test agent and running tests under a non-administrative account or under a service account is not supported.
- Set up environments to run continuous test tasks with your builds
- Testing in Continuous Integration and Continuous Deployment Workflows
- Deploy Azure Resource Group
- Azure File Copy
- Windows Machine File Copy
- PowerShell on Target Machines
- Visual Studio Test Agent Deployment
How do I configure a Machine Group for testing?
Choose the Manage link next to the Machines
parameter to open the Machine Group page where you can create and manage your Machine Groups. See Testing in Continuous Integration and Continuous Deployment Workflows for more details.
How do I create an Azure Resource Group for testing?
Where can I get more information about the Run Settings file?
Where can I get more information about overriding settings in the Run Settings file?
How can I customize code coverage analysis and manage inclusions and exclusions
Do I need a build agent?
You need at least one agent to run your build. Get an agent.
I can't select a default agent queue and I can't queue my build. How do I fix this?
I use Team Foundation Server on-premises and I don't see some of these features. Why not?
How can I submit feedback on this task?
Visit our UserVoice forum