FAQ for cloud-based load testing

These sections cover FAQ for cloud-based load testing to help you troubleshoot any issues or answer questions that you might have.

App Configuration

A: Any app that is available on the internet can be load tested using Visual Studio Team Services. Get started here.

If your app is behind a firewall because it is an internal app or your app has not been released yet, you can still use cloud-based load testing. To learn more, see Load testing applications behind a firewall using Trusted IP.

General

A: It can take anywhere from 5 seconds to 3 hours to get the permissions to run the load test in the cloud. If you had previously created the VSTS account, you may be able to run the load test right away.
A: No. There is currently no support for running load tests with mstest.
A: By default, they are stored in the local SQL Express database. SQL express works for storing the test results from a trial run. As you download more load test results, use SQL Server instead for better performance. To use SQL Server, follow these instructions.
A: Starting with Visual Studio 2013 Update 5 and Visual Studio 2015, you can select the test agent location when you configure your load test run. Choose a location from any supported Azure datacenter around the world. What if I’m using an earlier version of Visual Studio?

After your run finishes, your results are stored in the same location as your VSTS account.

If you’re using an earlier version of Visual Studio, the agent location is based on the location that you chose when you created your VSTS account.

VSTS Account Region
South Central US
West Europe
Test Agent Azure Datacenter
East US 2
West Europe

If your app is behind your firewall, you can still load test your app by configuring your firewall with Trusted IP. To request trusted IPs and learn how to use them when load testing internal apps, see Load testing applications behind a firewall using Trusted IP.

A: No. Only one load test run is hosted on the set of virtual machines where agents are hosted for that run.
A: The maximum number of cores for the load test agents for each run is 100 cores. If your test runs need more cores, you can run 10 load tests at the same time.

The minimum number of virtual users per agent core is 1. If your load test requires more agent cores OR fewer virtual users per agent core, please contact vsoloadtest@microsoft.com.

A: If your test run uses 25 or more virtual users per core, then VUMs = (max user load for your test run) * (test run duration in minutes).

If your test run uses fewer than 25 virtual users per core, then VUMs = (number of cores) * (25 virtual users per core) * (test run duration in minutes).

The minimum values used to calculate VUMs are 25 virtual users and 1 minute. If your test run values are smaller than the minimum values, then those values are rounded up to meet the minimums. For example, if your test run specifies 20 virtual users for 30 seconds, then your test will actually run with 25 virtual users for 1 minute = 25 VUMs, not 15 VUMs.

Also, test run duration is in minutes. So, if your test run duration is 5 minutes and 15 seconds, then that duration is rounded up to 6 minutes.

A minimum of 250 VUMs, including the test warm-up period, is deducted from your account for:

  • Completed runs, based on the full duration of the run
  • Aborted runs, based on the elapsed run duration

For runs that end in an error state, no VUMs are deducted from your account.

To check how many virtual user minutes that your VSTS account has used or has remaining, go to your VSTS account home page (https://{youraccount}.visualstudio.com).

A: Resource limits apply to each VSTS account. For each account, you receive 20,000 free virtual user minutes per month. If you need more virtual user minutes for your load testing, have your account owner get additional resources for your VSTS account.

If your free resource limits run out and you have not purchased additional resources, you will get a status message like this:

This run exceeds the maximum allowed usage for this month. The current usage for your account for this month (including runs in progress) is 8000 and the maximum allowed usage is 20000. To learn more about usage limits and how to modify them, refer to https://go.microsoft.com/fwlink/?LinkId=303976.

Configure Tests

A: Use a .csv file or an Excel spreadsheet to provide different values for cloud-based load testing. Using SQL Server is not currently supported. To learn how to supply these values to your test, go here.
A: The number of agents used by your run is based on your tests. If you get errors when you run your test, you might have to increase the number of agent cores. When you load test in VSTS using the Visual Studio IDE, you can change the number of agent cores.

IC666801

Agent Count (Total Cores)

What do the values mean?

  • 0: (Default) The number of cores is based on the number of virtual users that you specify for your test.
  • 1: Your test run will use 1 agent with 1 core.
  • 2 or more: Each agent will always use 2 cores. For example, if the value is 4, then you get 2 agents with 2 cores each. If the value is 3, you’ll still get 2 agents with 2 cores each. You won’t get 1 agent with 2 cores and 1 agent with just 1 core.

The number of agents also depends on your test mix (web performance test or unit test). If you have only web performance tests, then we suggest using 250 to 1000 virtual users for every 2 cores. If you have unit tests, the agent count depends on what your unit tests do. This means you will have to test if you have enough agents by running a shorter duration load test run or use goal-based load testing.

A: You can use deployment options and a Setup Script in test settings. You can add the .exe or other files to the Deployment window which you want to deploy on the Agent and using the Setup script you can install them on agents.

All the items deployed on the agents are copied to a directory on the agent. The location of the directory can be accessed by using %DeploymentDirectory% in the setup and cleanup script. For example, if I want to install WebDeploy on the agent machine I should add WebDeploy_x64_en-US.msi to Deployment window. Setup.cmd will look like %DeploymentDirectory%\WebDeploy_x64_en-US.msi /passive

Run and Monitor Load Tests

A: You can view the service status on the VSTS support portal (at the top of the page) and on our service blog. You can also subscribe to alerts for service status by following this post in our support forum.
A: The states for your load test run when you run with VSTS are:

  • In-Progress: The test run is currently running in the cloud.
  • Completed: The test run has completed successfully.
  • Aborted: The test run has been stopped by the user clicking the stop button. This state can also occur if there are issues related to your load test. For example, aborted can occur if there are issues in your test scripts.
  • Error: The test run has stopped due to an error with the service itself. For example, there might be an infrastructure issue in the service and it is unable to continue to run your test. This is not an issue caused by your load test or test scripts.
A: Due to a known issue, you must currently use this workaround:

  1. Start Notepad with administrator privileges
  2. Open devenv.exe.config file (this file is generally located at: “C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE”)
  3. Change the value of bindingRedirect to “8.0.0.0-14.0.0.0”
    <dependentAssembly>
        <assemblyIdentity name="Microsoft.VisualStudio.QualityTools.LoadTest" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
        <bindingRedirect oldVersion="8.0.0.0-14.0.0.0" newVersion="12.0.0.0"/>
    </dependentAssembly>

Troubleshooting

A: If the web test recorder is not enabled when you try and record your web test with Internet Explorer 11, go here to resolve the issue.
A: Status messages and test errors are reported while your load test runs. Status messages give you details about the load test run itself, such as when a connection to the results database was lost. Test errors relate to the test. View both these messages from the Details tab of the progress graphs.

IC696061

A: If the error states that the connection’s current state is closed, you can set the amount of time a connection waits before timing out. Set the ConnectTimeout or Connection Timeout keywords in the connection string. Do not set a value of 0 as a timeout in a ConnectionString because the connection will keep trying to connect indefinitely.
A: If this happens, you need to take the Visual Studio Ultimate Product Key from your MSDN subscription and use the “Change my Product License” option on the Product Information page. You need to do this on every machine where you want to run load tests using VSTS. Visit this site to get the product key.
A: Starting on 26th November 2014, you must add the version information to your REST API calls. If your call fails with a VssVersionNotSpecifiedException exception, you must include ?api-version=1.0-preview.1 in your REST API calls. Follow the instructions here to do this.
A: Test names in load tests are converted to lower case when the test runs using VSTS. Any string match done on a test name by user code should ignore the case or convert test names to lower case.
A: Use a text editor to edit devenv.exe.config. This file is generally located in “C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE”.

    1. Add the following line inside the <appSettings> section:
      <add key="ElsClientLogLevel" value="XXX"/>

      Where XXX can be any of the following:

      • all – logs all messages
      • off – stops logging any messages
      • critical – only logs critical messages
      • error – only logs error and critical messages
      • warning – logs error, critical and warning messages (default)
      • information – logs error, critical, warning and info messages
      • verbose – logs error, critical, warning, info and verbose messages

 

  1. Add the following section to the bottom of the devenv.exe.config file, just above the closing tag. You can specify the path for the log file by changing the initializeData value.
    <system.diagnostics>
     <trace autoflush="true" indentsize="4">
     <listeners>
     <add name="myListener" type="System.Diagnostics.TextWriterTraceListener" initializeData="d:\VSTestHost.log"/>
     </listeners>
     </trace>
    
     <switches>
     <!-- You must use integral values for "value": 0 = off, 1 = error, 2 = warn, 3 = info, 4 = verbose.-->
     <add name="EqtTraceLevel" value="4" />
     </switches>
     </system.diagnostics>
  2. Restart Visual Studio 2013 and reproduce the issue. You can then review the log file or share it with support. The log file is located here: %Temp%\ELSClient\.
A: For Visual Studio 2013 Update 4 and later versions, the default value for the TimingDetailsStorage property has been changed from AllIndividualDetails to None. If you want to collect the individual timings, you must specifically set TimingDetailsStorage property to be AllIndividualDetails. Go here for more details.

Errors

A: If you get one of these errors:

  • VS1550064
  • VS1550072
  • VS1550078
  • VS1550081
  • VS1550082
  • VS1550083

Contact VSTS support. You will need to give them your test run id.

A: If you manually edit the .loadtest xml file, this can result in this error:

  • VS1550084

Open the file and revert any changes that you added. Rerun the load test and the run should complete successfully.

A: If you manually edit the .loadtest xml file, this can result in these errors:

  • VS1550026
  • VS1550027

Open the file and revert any changes that you added. Rerun the load test and the run should complete successfully.

A: If you close the load test wizard without completing it, this can cause this error:

  • VS1550030

To fix this issue, create another load test and delete the one that failed to run.

A: These errors are generally due to issues with the cloud-based load testing service. Just try and run your load test again. If the issue persists, contact VSTS support. You will need to give them your test run id.
A: Go here to find information about other errors and their resolutions, where applicable.

Use the cloud service

Get started for free

Host it yourself

Download trial