Summary: This document outlines the performance test best practices in Strategy Integrity Manager 9.4.x-10.x.
The results of a performance test can be affected by many factors. The following best practices can help ensure that users get the most accurate results from a performance test:
- Performance comparison tests should be run as single-project integrity tests. This reduces the load on Integrity Manager and ensures that the recorded times are as accurate as possible.
To compare performance on two Intelligence Servers, Strategy recommends following the procedure below:
- Perform a single project test against one project, saving the performance results.
- Perform a single project test against the second project, saving the performance results.
- Compare the two performance results in a baseline-versus-baseline test.
- Wait until the performance test is complete before attempting to view the results of the test in Integrity Manager. Otherwise the increased load on the Integrity Manager machine can cause the recorded times to be increased for reasons not related to Intelligence Server performance.
- If using a baseline-versus-baseline test or a baseline-versus-project test, make sure that the tests have processed the reports/documents in the same formats. Execution times are not recorded for each format, only for the aggregate generation of the selected formats. Thus, comparing a baseline of SQL and Graph data against a test of only SQL data is likely to give inaccurate results.
- If the Use Report Cache setting is selected on the Select Execution Settings page of the Integrity Manager wizard, make sure that a valid cache exists for each report or document to be tested. Otherwise the first execution cycle of each report/document takes longer than the subsequent cycles, because it must generate the cache for the other cycles to use. One way to ensure that a cache exists for each report/document is to run a single-project integrity test of each report before you run the performance test.
Note: This setting only applies to reports. Integrity Manager does not use document caches.
- On the Select Execution Settings page of the Integrity Manager wizard, make sure Concurrent Jobs is set to 1. This causes Intelligence Server to run only one report or document at a time, providing the most accurate benchmark results for that Intelligence Server.
- The Cycles setting on the Select Processing Options page indicates how many times each report or document is executed. A high value for this setting can dramatically increase the execution time of your test, particularly if you are running many reports or documents, or several large reports.
