FAQ for cloud-based load testing

These sections cover FAQ for cloud-based load testing to help you troubleshoot any issues or answer questions that you might have.

App Configuration

Q: Can I use cloud-based load testing to test any app, even if it's behind a firewall?

A: Any app that is available on the internet can be load tested using Visual Studio Team Services. Get started here.

If your app is behind a firewall because it is an internal app or your app has not been released yet, you can still use cloud-based load testing. To learn more, see Load testing applications behind a firewall using Trusted IP.


Q: How long do I have to wait until I can run my load test after creating a Visual Studio Team Services account?
A: It can take anywhere from 5 seconds to 3 hours to get the permissions to run the load test in the cloud. If you had previously created the Visual Studio Team Services account, you may be able to run the load test right away.
Q: Can I use mstest to run load tests with Visual Studio Team Services?

A: No. There is currently no support for running load tests with mstest.

Q: Where are the load test results stored after I download them to the client machine from the cloud?
A: By default, they are stored in the local SQL Express database. SQL express works for storing the test results from a trial run. As you download more load test results, use SQL Server instead for better performance. To use SQL Server, follow these instructions.
Q: Where are the test agents used for my load test runs located?

A: Starting with Visual Studio 2013 Update 5 and Visual Studio 2015, you can select the test agent location when you configure your load test run. Choose a location from any supported Azure datacenter around the world. What if I'm using an earlier version of Visual Studio?

After your run finishes, your results are stored in the same location as your Visual Studio Team Services account.

If you're using an earlier version of Visual Studio, the agent location is based on the location that you chose when you created your Visual Studio Team Services account.

Visual Studio Team Services Account Region             Test Agent Azure Datacenter
South Central US East US 2
West Europe West Europe

If your app is behind your firewall, you can still load test your app by configuring your firewall with Trusted IP. To request trusted IPs and learn how to use them when load testing internal apps, see Load testing applications behind a firewall using Trusted IP.

Q: Are the virtual machines for my agents shared with any other load test runs?
A: No. Only one load test run is hosted on the set of virtual machines where agents are hosted for that run.
Q: What are the maximum cores for the agents and minimum number of virtual users per core?

A: The maximum number of cores for the load test agents for each run is 100 cores. If your test runs need more cores, you can run 10 load tests at the same time.

The minimum number of virtual users per agent core is 1. If your load test requires more agent cores OR fewer virtual users per agent core, please contact vsoloadtest@microsoft.com.

Q: What are virtual user minutes (VUMs)? How many minutes will my load test run use?

A: If your test run uses 25 or more virtual users per core, then VUMs = (max user load for your test run) * (test run duration in minutes).

If your test run uses fewer than 25 virtual users per core, then VUMs = (number of cores) * (25 virtual users per core) * (test run duration in minutes).

The minimum values used to calculate VUMs are 25 virtual users and 1 minute. If your test run values are smaller than the minimum values, then those values are rounded up to meet the minimums. For example, if your test run specifies 20 virtual users for 30 seconds, then your test will actually run with 25 virtual users for 1 minute = 25 VUMs, not 15 VUMs.

Also, test run duration is in minutes. So, if your test run duration is 5 minutes and 15 seconds, then that duration is rounded up to 6 minutes.

A minimum of 250 VUMs, including the test warm-up period, is deducted from your account for:

  • Completed runs, based on the full duration of the run
  • Aborted runs, based on the elapsed run duration

For runs that end in an error state, no VUMs are deducted from your account.

To check how many virtual user minutes that your Visual Studio Team Services account has used or has remaining, go to your Visual Studio Team Services account home page (https://{youraccount}.visualstudio.com).

Q: What are the current resource limits for running load tests using Visual Studio Team Services?

A: Resource limits apply to each Visual Studio Team Services account. For each account, you receive 20,000 free virtual user minutes per month. If you need more virtual user minutes for your load testing, have your account owner get additional resources for your Visual Studio Team Services account.

If your free resource limits run out and you have not purchased additional resources, you will get a status message like this:

This run exceeds the maximum allowed usage for this month. The current usage for your account for this month (including runs in progress) is 8000 and the maximum allowed usage is 20000. To learn more about usage limits and how to modify them, refer to https://go.microsoft.com/fwlink/?LinkId=303976.

Configure Tests

Q: How do I provide different values to the same test?
A: Use a .csv file or an Excel spreadsheet to provide different values for cloud-based load testing. Using SQL Server is not currently supported. To learn how to supply these values to your test, go here.
Q: How do I calculate the number of agents to use for my run?

A: The number of agents used by your run is based on your tests. If you get errors when you run your test, you might have to increase the number of agent cores. When you load test in Visual Studio Team Services using the Visual Studio IDE, you can change the number of agent cores.

Visual Studio; open load test and choose Run Settings to edit Agent Count in the Properties window 

Agent Count (Total Cores)

What do the values mean?

  • 0: (Default) The number of cores is based on the number of virtual users that you specify for your test.
  • 1: Your test run will use 1 agent with 1 core.
  • 2 or more: Each agent will always use 2 cores. For example, if the value is 4, then you get 2 agents with 2 cores each. If the value is 3, you'll still get 2 agents with 2 cores each. You won't get 1 agent with 2 cores and 1 agent with just 1 core.

The number of agents also depends on your test mix (web performance test or unit test). If you have only web performance tests, then we suggest using 250 to 1000 virtual users for every 2 cores. If you have unit tests, the agent count depends on what your unit tests do. This means you will have to test if you have enough agents by running a shorter duration load test run or use goal-based load testing.

Q: How do I install certificates/software on agents that run my load tests in the cloud?

A: You can use deployment options and a Setup Script in test settings. You can add the .exe or other files to the Deployment window which you want to deploy on the Agent and using the Setup script you can install them on agents.

All the items deployed on the agents are copied to a directory on the agent. The location of the directory can be accessed by using %DeploymentDirectory% in the setup and cleanup script. For example, if I want to install WebDeploy on the agent machine I should add WebDeploy_x64_en-US.msi to Deployment window. Setup.cmd will look like %DeploymentDirectory%\WebDeploy_x64_en-US.msi /passive

Run and Monitor Load Tests

Q: How can I check the status of the load testing service?

A: You can view the service status on the Visual Studio Team Services support portal (at the top of the page) and on our service blog. You can also subscribe to alerts for service status by following this post in our support forum.

Q: What are the possible load test run states?

A: The states for your load test run when you run with Visual Studio Team Services are:

  • In-Progress: The test run is currently running in the cloud.
  • Completed: The test run has completed successfully.
  • Aborted: The test run has been stopped by the user clicking the stop button. This state can also occur if there are issues related to your load test. For example, aborted can occur if there are issues in your test scripts.
  • Error: The test run has stopped due to an error with the service itself. For example, there might be an infrastructure issue in the service and it is unable to continue to run your test. This is not an issue caused by your load test or test scripts.
Q: How should I view test logs after downloading the test results locally?

A: Due to a known issue, you must currently use this workaround:

  1. Start Notepad with administrator privileges
  2. Open devenv.exe.config file (this file is generally located at: "C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE")
  3. Change the value of bindingRedirect to ""

        <assemblyIdentity name="Microsoft.VisualStudio.QualityTools.LoadTest" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
        <bindingRedirect oldVersion="" newVersion=""/>


Q: How can I record a web performance test with Internet Explorer 11?

A: If the web test recorder is not enabled when you try and record your web test with Internet Explorer 11, go here to resolve the issue.

Q: What should I do if Visual Studio stops responding when I try to run a load test in the cloud?
A: Go here for a solution to this issue.
Q: How can I view errors and warnings that happen when my load test is running in the cloud?

A: Status messages and test errors are reported while your load test runs. Status messages give you details about the load test run itself, such as when a connection to the results database was lost. Test errors relate to the test. View both these messages from the Details tab of the progress graphs.

From Visual Studio, click Details from the tab that shows your load test running in the cloud 

Q: I get an error when I try to import downloaded test results. What should I do?
A: If the error states that the connection's current state is closed, you can set the amount of time a connection waits before timing out. Set the ConnectTimeout or Connection Timeout keywords in the connection string. Do not set a value of 0 as a timeout in a ConnectionString because the connection will keep trying to connect indefinitely.
Q: Why can't I use more than 250 virtual users or plug-ins when I have Visual Studio Enterprise?
A: If this happens, you need to take the Visual Studio Ultimate Product Key from your MSDN subscription and use the "Change my Product License" option on the Product Information page. You need to do this on every machine where you want to run load tests using Visual Studio Team Services. Visit this site to get the product key.
Q: Why have the REST API calls that I use stopped working?
A: Starting on 26th November 2014, you must add the version information to your REST API calls. If your call fails with a VssVersionNotSpecifiedException exception, you must include ?api-version=1.0-preview.1 in your REST API calls. Follow the instructions here to do this.
Q: I've noticed that user code fails to execute if it depends on the test names. Are test names changed when run against the service?
A: Test names in load tests are converted to lower case when the test runs using Visual Studio Team Services. Any string match done on a test name by user code should ignore the case or convert test names to lower case.
Q: How do I enable client-side logs to help troubleshoot issues with load tests run in the cloud?

A: Use a text editor to edit devenv.exe.config. This file is generally located in “C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE".

  1. Add the following line inside the <appSettings> section:

    <add key="ElsClientLogLevel" value="XXX"/>

    Where XXX can be any of the following:

    • all - logs all messages
    • off - stops logging any messages
    • critical - only logs critical messages
    • error - only logs error and critical messages
    • warning - logs error, critical and warning messages (default)
    • information - logs error, critical, warning and info messages
    • verbose - logs error, critical, warning, info and verbose messages
  2. Add the following section to the bottom of the devenv.exe.config file, just above the closing tag. You can specify the path for the log file by changing the initializeData value.

    <trace autoflush="true" indentsize="4">
    <add name="myListener" type="System.Diagnostics.TextWriterTraceListener" initializeData="d:\VSTestHost.log"/>

    <!-- You must use integral values for "value": 0 = off, 1 = error, 2 = warn, 3 = info, 4 = verbose.-->
    <add name="EqtTraceLevel" value="4" />
  3. Restart Visual Studio 2013 and reproduce the issue. You can then review the log file or share it with support. The log file is located here: %Temp%\ELSClient\.

Q: Why don't I see the individual timing values in the Load Tests Results Store?
A: For Visual Studio 2013 Update 4 and later versions, the default value for the TimingDetailsStorage property has been changed from AllIndividualDetails to None. If you want to collect the individual timings, you must specifically set TimingDetailsStorage property to be AllIndividualDetails. Go here for more details.


Q: If I get an error that my test execution run failed, what should I do?

A: If you get one of these errors:

  • VS1550064
  • VS1550072
  • VS1550078
  • VS1550081
  • VS1550082
  • VS1550083

Contact Visual Studio Team Services support. You will need to give them your test run id.

Q: If I get an error that my run was aborted because the .loadtest xml file could not be parsed, what should I do?

A: If you manually edit the .loadtest xml file, this can result in this error:

  • VS1550084

Open the file and revert any changes that you added. Rerun the load test and the run should complete successfully.

Q: If I get an error that too many applications or counters have been selected to run for my load test, what should I do?

A: If you manually edit the .loadtest xml file, this can result in these errors:

  • VS1550026
  • VS1550027

Open the file and revert any changes that you added. Rerun the load test and the run should complete successfully.

Q: If I get an error that no active load test settings were found in my load test, what should I do?

A: If you close the load test wizard without completing it, this can cause this error:

  • VS1550030

To fix this issue, create another load test and delete the one that failed to run.

Q: If my load test run gets an error when it starts or is aborted during the run, what should I do?
A: These errors are generally due to issues with the cloud-based load testing service. Just try and run your load test again. If the issue persists, contact Visual Studio Team Services support. You will need to give them your test run id.
Q: Where can I find information about other errors?

A: Go here to find information about other errors and their resolutions, where applicable.