FAQ for cloud-based load testing

These sections cover FAQ for cloud-based load testing to help you troubleshoot any issues or answer questions that you might have.

App Configuration

Q: Can I use cloud-based load testing to test any app, even if it's behind a firewall?

A: Any app that is available on the internet can be load tested using Visual Studio Online. Get started here.

If your app is behind a firewall because it is an internal app or your app has not been released yet, you can still use cloud-based load testing. Contact vsoloadtest@microsoft.com to learn how to do this.

General

Q: Where are the test agents used for my load test runs located?

A: The location of the agents will be based on the location that you chose when you created your Visual Studio Online account.

Visual Studio Online Account Region             Test Agent Azure Datacenter
South Central US East US 2
West Europe West Europe

The Microsoft Azure Datacenter IP ranges can be downloaded from here.

Q: Where are the load test results stored after I download them to the client machine from the cloud?
A: By default, they are stored in the local SQL Express database. SQL express works for storing the test results from a trial run. As you download more load test results, use SQL Server instead for better performance. To use SQL Server, follow these instructions.
Q: Are the virtual machines for my agents shared with any other load test runs?
A: No. Only one load test run is hosted on the set of virtual machines where agents are hosted for that run.
Q: Can I use mstest to run load tests with Visual Studio Online?

A: No. There is currently no support for running load tests with mstest.

Q: Where are the agents for my load test run located? Can I choose that location?
A: All load testing agents are currently located in the Microsoft Azure East US data center. This location cannot be changed.
Q: What are VUserMinutes? How many will my load test run use?

A: VUserMinutes = (Maximum user load for the run) * (Duration of the run in minutes).

The minimum values used for calculating VUserMinutes are 25 VUsers and 1 minute. If your run values are smaller than the minimums, then the value will be rounded up to the minimum value. For example, if your load test is set up to run with 100 VUsers for 30 seconds, then the VUserMinutes will be calculated with 1 minute for the duration: 100 (VUsers) * 1 (minute) = 100. The same rule applies to VUser count calculation.

If your run duration is 5 minutes and 15 seconds, then the run duration will be rounded up to the next integer value. In this example, the duration will be 6 minutes.

VUserMinutes are deducted from your account for:

  • Any runs that are in a completed state
  • Any runs that are in an aborted state, based on the actual run duration - a minimum of 250 VUserMinutes will be deducted (including the warm-up period)

For runs that end in an error state, no VUserMinutes will be deducted from your account.

View how many VUserMinutes have already been used by your account from your account home page (https://fabrikam-prime.visualstudio.com). The number of minutes remaining is displayed in the resource usage section. 

Q: What are the current resource limits for running load tests using Visual Studio Online?

A: Resource limits apply to each Visual Studio Online account. For each account, you receive 20,000 free virtual user minutes per month. If you need more virtual user minutes for your load testing, have your account owner get additional resources for your Visual Studio Online account.

If your free resource limits run out and you have not purchased additional resources, you will get a status message like this:

This run exceeds the maximum allowed usage for this month. The current usage for your account for this month (including runs in progress) is 8000 and the maximum allowed usage is 20000. To learn more about usage limits and how to modify them, refer to http://go.microsoft.com/fwlink/?LinkId=303976.

Q: What are the maximum cores for the agents and minimum number of users per core?

A: The maximum number of cores for the load test agents for each run is 20 cores.

Minimum VUsers per agent core is 25. This value is calculated by dividing the ‘Maximum User Load’ by ‘Total Agent cores’ in your load run. You need to have a minimum of 25 VUsers per agent core in your run.

If your load test has a requirement of a greater number of agent cores OR lesser number of ‘VUsers per agent core’ please contact Microsoft support for help.

Configure Tests

Q: How do I provide different values to the same test?
A: Use a .csv file or an Excel spreadsheet to provide different values for cloud-based load testing. Using SQL Server is not currently supported. To learn how to supply these values to your test, go here.
Q: How long do I have to wait until I can run my load test after creating a Visual Studio Online account?
A: It can take anywhere from 5 seconds to 3 hours to get the permissions to run the load test in the cloud. If you had previously created the Visual Studio Online account, you may be able to run the load test right away.
Q: How do I calculate the number of agents that should be used for my run?

A: The number of agents used by your run is based on your tests. If you get errors when you run your test, you may need to increase the number of agents. The ‘Agent Count’ property in Run Settings sets the number of agent cores you want to use to run your tests on Visual Studio Online.

Agent Count (Total Cores)

The default value is 0.

What do the values mean?

  • 0: Total cores allocated is based on the number of virtual users (Vusers) for your test.
  • 1: Your test run will be allocated one agent with one core
  • 2 or greater: Agents are always allocated with 2 cores. For example, if the value is 4 then 2 agents each with 2 cores are allocated to your run. An agent with just one core will not be allocated. So if the value is 3, then 2 agents each with 2 cores will be allocated.

There is a limit to the number of virtual users per core which a test run can have. The ratio of virtual users/core needs to be a minimum of 25 for the load test to be run on the service. For example, if you were running a test with 6 virtual users and 2 agent cores, this makes the ratio of virtual users/core as 6/2 = 3, which is clearly less than 25. So, you will not be able to execute this test in the cloud. This ratio is not applied if the Agent Count is set to the default of 0.

If this ratio is incorrect, you will see this status message: "A load test cannot be executed with less than 25 virtual-users/core for one run. Reduce the agent count or increase the virtual users in your load test and run the test again".

The number of test agents required for your load test are dependent on the kind of test mixes (web performance test or unit test) that you include as part of the load test. If you have only web performance tests, then the recommendation is to have 250 to 1000 virtual users (Vusers) per 2 agent cores. For unit tests in the load test, we don’t provide any recommendations. You might want to test if the number of test agents will be sufficient by running a shorter duration load test run first.

Q: How do I change the number of agents used in my run?

A: Edit the Agent Count (Total Cores) property in Run Settings to change the number of agents. The default value of 0 allocates one agent with one core for your run.

Visual Studio; open load test and choose Run Settings to edit Agent Count in the Properties window 

Q: How do I install certificates/software on agents that run my load tests in the cloud?

A: You can use deployment options and a Setup Script in test settings. You can add the .exe or other files to the Deployment window which you want to deploy on the Agent and using the Setup script you can install them on agents.

All the items deployed on the agents are copied to a directory on the agent. The location of the directory can be accessed by using %DeploymentDirectory% in the setup and cleanup script. For example, if I want to install WebDeploy on the agent machine I should add WebDeploy_x64_en-US.msi to Deployment window. Setup.cmd will look like %DeploymentDirectory%\WebDeploy_x64_en-US.msi /passive

Run and Monitor Load Tests

Q: How can I check the status of the load testing service?

A: You can view the service status on the Visual Studio Online support portal (at the top of the page) and on our service blog. You can also subscribe to alerts for service status by following this post in our support forum.

Q: What are the possible load test run states?

A: The states for your load test run when you run with Visual Studio Online are:

  • In-Progress: The test run is currently running in the cloud.
  • Completed: The test run has completed successfully.
  • Aborted: The test run has been stopped by the user clicking the stop button. This state can also occur if there are issues related to your load test. For example, aborted can occur if there are issues in your test scripts.
  • Error: The test run has stopped due to an error with the service itself. For example, there might be an infrastructure issue in the service and it is unable to continue to run your test. This is not an issue caused by your load test or test scripts.
Q: How should I view test logs after downloading the test results locally?

A: Due to a known issue, you must currently use this workaround:

  1. Start Notepad with administrator privileges
  2. Open devenv.exe.config file (this file is generally located at: "C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE")
  3. Change the value of bindingRedirect to "8.0.0.0-14.0.0.0"

    <dependentAssembly>
        <assemblyIdentity name="Microsoft.VisualStudio.QualityTools.LoadTest" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
        <bindingRedirect oldVersion="8.0.0.0-14.0.0.0" newVersion="12.0.0.0"/>
    </dependentAssembly>

Troubleshooting

Q: How can I record a web performance test with Internet Explorer 11?

A: If the web test recorder is not enabled when you try and record your web test with Internet Explorer 11, go here to resolve the issue.

Q: What should I do if Visual Studio stops responding when I try to run a load test in the cloud?
A: Go here for a solution to this issue.
Q: How can I view errors and warnings that happen when my load test is running in the cloud?

A: Status messages and test errors are reported while your load test runs. Status messages give you details about the load test run itself, such as when a connection to the results database was lost. Test errors relate to the test. View both these messages from the Details tab of the progress graphs.

From Visual Studio, click Details from the tab that shows your load test running in the cloud 

Q: I get an error when I try to import downloaded test results. What should I do?
A: If the error states that the connection's current state is closed, you can set the amount of time a connection waits before timing out. Set the ConnectTimeout or Connection Timeout keywords in the connection string. Do not set a value of 0 as a timeout in a ConnectionString because the connection will keep trying to connect indefinitely.
Q: Why can't I use more than 250 virtual users or plug-ins when I have Visual Studio Ultimate?
A: If this happens, you need to take the Visual Studio Ultimate Product Key from your MSDN subscription and use the "Change my Product License" option on the Product Information page. You need to do this on every machine where you want to run load tests using Visual Studio Online. Visit this site to get the product key.
Q: Why have the REST API calls that I use stopped working?
A: Starting on 26th November 2014, you must add the version information to your REST API calls. If your call fails with a VssVersionNotSpecifiedException exception, you must include ?api-version=1.0-preview.1 in your REST API calls. Follow the instructions here to do this.
Q: I've noticed that user code fails to execute if it depends on the test names. Are test names changed when run against the service?
A: Test names in load tests are converted to lower case when the test runs using Visual Studio Online. Any string match done on a test name by user code should ignore the case or convert test names to lower case.
Q: How do I enable client-side logs to help troubleshoot issues with load tests run in the cloud?

A: Use a text editor to edit devenv.exe.config. This file is generally located in “C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE".

  1. Add the following line inside the <appSettings> section:

    <add key="ElsClientLogLevel" value="XXX"/>

    Where XXX can be any of the following:

    • all - logs all messages
    • off - stops logging any messages
    • critical - only logs critical messages
    • error - only logs error and critical messages
    • warning - logs error, critical and warning messages (default)
    • information - logs error, critical, warning and info messages
    • verbose - logs error, critical, warning, info and verbose messages
  2. Add the following section to the bottom of the devenv.exe.config file, just above the closing tag. You can specify the path for the log file by changing the initializeData value.

                          <system.diagnostics>
    <trace autoflush="true" indentsize="4">
    <listeners>
    <add name="myListener" type="System.Diagnostics.TextWriterTraceListener" initializeData="d:\VSTestHost.log"/>
    </listeners>
    </trace>

    <switches>
    <!-- You must use integral values for "value": 0 = off, 1 = error, 2 = warn, 3 = info, 4 = verbose.-->
    <add name="EqtTraceLevel" value="4" />
    </switches>
    </system.diagnostics>
  3. Restart Visual Studio 2013 and reproduce the issue. You can then review the log file or share it with support. The log file is located here: %Temp%\ELSClient\.

Errors

Q: If I get an error that my test execution run failed, what should I do?

A: If you get one of these errors:

  • VS1550064
  • VS1550072
  • VS1550078
  • VS1550081
  • VS1550082
  • VS1550083

Contact Visual Studio Online support. You will need to give them your test run id.

Q: If I get an error that my run was aborted because the .loadtest xml file could not be parsed, what should I do?

A: If you manually edit the .loadtest xml file, this can result in this error:

  • VS1550084

Open the file and revert any changes that you added. Rerun the load test and the run should complete successfully.

Q: If I get an error that too many applications or counters have been selected to run for my load test, what should I do?

A: If you manually edit the .loadtest xml file, this can result in these errors:

  • VS1550026
  • VS1550027

Open the file and revert any changes that you added. Rerun the load test and the run should complete successfully.

Q: If I get an error that no active load test settings were found in my load test, what should I do?

A: If you close the load test wizard without completing it, this can cause this error:

  • VS1550030

To fix this issue, create another load test and delete the one that failed to run.

Q: If my load test run gets an error when it starts or is aborted during the run, what should I do?
A: These errors are generally due to issues with the cloud-based load testing service. Just try and run your load test again. If the issue persists, contact Visual Studio Online support. You will need to give them your test run id.