Load testing in the cloud
Run performance tests on your app when you expect an increased user load to check your response times. No need to use resources and set up your own machines to create this load. You can use cloud-based load testing to provide virtual machines that generate the load of many users accessing your web site at the same time. You get access to the resources you need to run performance tests when you need them, including 20,000 free virtual user minutes per month.
Drill down further into your app's performance with real-world load tests:
- Record the specific steps in your app for your test
- Add the estimated time users spend thinking into your test
- Run tests for different tasks in your app at the same time
- Choose the mix of browsers to simulate your users better
- Get advanced reports that integrate performance data from Application Insights
Your web site must be publicly available on the internet for cloud-based load testing using Visual Studio Online to access it.
If you already have existing load tests, or you know how to create them using Visual Studio, jump ahead to configure your project to run load tests in the cloud.
Download and configure a sample load test
Use Visual Studio 2013 Ultimate to create and run load tests in the cloud. If you don't have this installed on your machine, or you have not installed Visual Studio 2013 Update 3, download and install from here.
Extract the files and open the GettingStartedWithLoadTesting.sln.
Open the file SampleWebTest.webtest and replace the URL to use a web page for your app.
Save the solution.
Configure your project to run load tests in the cloud
If you are using the sample project, jump ahead to connect to your team project.
If you have existing load tests that you created using Visual Studio 2010 or Visual Studio 2012. Simply open your existing project using Visual Studio 2013 first.
Edit the test settings file to configure your project to run your tests using Visual Studio Online. (Only load tests can be run in the cloud.)
You do not have to connect to Visual Studio Online with your Visual Studio account until you are ready to run the load test.
Connect to your team project
Before you can run your load tests in the cloud, you must connect to your team project in Visual Studio Online. (If your team doesn't already have a Visual Studio account or a team project, create them.)
Connect to your team project from Visual Studio Team Explorer.
Display the list of available team projects.
If you have not yet connected to your Visual Studio Online account, then you need to add it to your list.
Enter the name for your account -[your account].visualstudio.com.
If you are prompted to sign in to your Microsoft account, go ahead and do that.
Select the server from the list and then select your team project. Now you can connect.
Run and analyze your load test
Open the load test you want to run from your solution in Solution Explorer.
Run the load test in the cloud using Visual Studio Online.
You can see when your test is in the queue for the service.
When your load test is ready to be run, the test status changes to: Acquiring resources. For a large test run this can take up to ten minutes while the service does the hard work of setting up machines and agents for you.
When the test is running, you can see the performance graph. Choose details to see any errors, warnings or other information related to your load test run.
You can also view the throughput graph.
After the load test is finished, you can download the report.
When it has downloaded, you can view the report.
The results for the completed test includes performance counter data, threshold violations, and error information.
Choose the detail view. By analyzing the step load pattern for users, you can identify the user count where your performance fails to meet your requirements.
Fix any performance issues in your application’s code and rerun the test. You can also refine your test to better simulate real-world loads by specifying various web performance test properties, load test scenario properties and run settings properties.
View your load test runs at any time
Use load test manager to view past load test runs or currently running load tests. (You must have a web performance and load test project open to do this.)
Now you can view any load test runs started by any member of your team.
You can filter out the test runs that you see by state, date, or user who created the test run.
Get performance data from Application Insights
When you run your load tests for your app using Visual Studio Online, you can use Application Insights to retrieve performance data as well. By correlating load and performance data together, you get more insight into any performance issues that you find. Then you can run a quick root cause analysis using Application Insights, too.
To retrieve data from Application Insights when you run your load tests, install Visual Studio 2013 Update 4.
If you haven't set up monitoring for your app with Application Insights, do that first. You must use the same Microsoft account to monitor your app and run your load test with Visual Studio Online.
Open your load test in Visual Studio.
Add applications to get performance data from Application Insights.
Select the applications that you want to monitor and the counters that you want to view when your load test runs.
When your load test runs, you can view the performance data from Application Insights. There may be a delay of a few minutes before this data is displayed.
Application counters are correlated with the user load so that you can understand what issues may be causing any performance problems that you find.
Go to Application Insights if you need to do a more detailed analysis of any performance issue or to do a quick root cause analysis.
Q&A (Cloud-based load testing)
Q:Can I use cloud-based load testing to test any internal web site?
A:No. You cannot use a web site that is only available internally to your company. For example, an internal staging or preproduction deployment. However, you can open up the endpoint for your web site for limited IP addresses or users to work with cloud-based load testing. Or you can run a load test locally using Visual Studio.
Q:What are test agents and why do I need them?
A:Test agents simulate the load against your web site by creating the virtual users. You might need to use more agents depending on your load test and the number of virtual users you require. Increase the number of agents if you are getting status messages that the agent aborted due to load, or if the downloaded results show high CPU utilization for an agent. You cannot exceed an agent count (total cores) of 20 for a load test run.
The ratio of virtual users to agents must be 25 or greater for the load test to run in the cloud. For example, if you run a test with 40 virtual users and 2 agents, the ratio is 20. This is too low, so increase the number of virtual users or reduce the number of agents. If you use the default value of 0, this ratio is not applied.
Q:Are there any limits on virtual users when running cloud-based load tests?
A:Yes. The limit is 20,000 virtual user minutes per month. If you need more virtual user minutes for your load testing, have your account owner get additional resources for your Visual Studio Online account. The account will be charged for any virtual user minutes that you use above this limit each month.
Q:How can I run load tests locally and in the cloud from the same project?
A:You can have multiple test settings in a project. Add another test settings to the solution items folder. Now you can use one to run your tests locally and the other to run your load tests in the cloud. To switch between them, use the context menu to set the appropriate test settings to be active.
Q:Can load tests use other test types in their test mix besides web performance tests?
A:Yes, you can include unit tests and coded web tests. But coded UI tests are not supported with cloud-based load testing.
Q:Can I debug a load test while it's running in the cloud?
A:No. You need to run a load test locally to debug it.
Q:Can virtual users simulate pausing between test steps?
A:Yes, you can specify think times. Choose a scenario in your load test and edit the think time in the Properties view.
Q:When I download my load test report, where is it stored?
A:There is a local SQL express database where the downloaded reports are stored. You can change the default location if you want. Change the location for each user to the same database if you want to store all the reports together.
Q:Are there any load testing features that are not supported when you run a load test using cloud-based load testing?
A:These features are not currently supported:
- Goal-based load patterns
- Iteration-based runs
- Network mix property
- Warm up and cool down time
- Agent to Use in test settings - use the core count property instead
- SQL Trace properties in run settings
- IP switching
Q:Having problems running your load tests in the cloud?
A:Check out the latest troubleshooting advice.
Q:Where do I get more information about simulating real-world loads?
A:Find out more about how to specify web performance test properties, load test scenario properties and run settings properties.
Q:Can I learn more about cloud-based load testing?
A:Watch this video.
Q&A (Get performance data from Application Insights)
Q:Can I view data from other application monitoring tools when I run load tests in the cloud?
Q:Can I increase the frequency used to collect the counters?
A:No. This is currently a fixed frequency of 1 minute.
Q:I don't see any counters even after waiting a few minutes. What's wrong?
A:Go to Application Insights and check that you can view performance data for the application there. If there is data collected there, report your issue to firstname.lastname@example.org. If there is no data, go here for troubleshooting help with Application Insights.