Variance control

Both iOS and Android performance analysis is designed to give you an accurate profiling of the performance of your app, and in the case of a performance comparison, determine if there was a significant change. This is done by running a test (i.e. launching your app or running a UI test) repeatedly until there are enough samples to give a statistically sound result.

Ideally, the code executed in each test would be the same, but many sources of variance can cause different code paths to be taken in each test iteration. For example, you might have some code that generates a random number and uses the result to decide what to do. This kind of code won't work well with performance testing, as it will perform differently over multiple runs even for one build of the app.

To ensure performance testing works with most apps, many sources of variances are automatically controlled by Emerge. For example, any network request made by the app is automatically cached and replayed, so a given request always produces the same response. The disk state is also reset between tests, so if the app writes to a file during a test, that file is restored to its initial state before the test runs again.

As a general rule of thumb, performance testing aims to control everything about the app's environment, so the only changes measured will be changes from within the uploaded app. This means a single upload will always take the same code paths while running in performance analysis.

Suppose you're trying to test something that is outside of the app's control, such as the performance two different users would see for the same app. In that case, the additional user data should be included in two separate app uploads (such as in a configuration plist on iOS or with BuildConfig settings on Android). These two different uploads can then be compared.

Network Replay

To prevent different network responses from causing invalid performance test results, all HTTP/HTTPS requests are intercepted and controlled. Base and head apps receive the same network state for each test iteration. Network Replay operates in three modes - disabled, recording, and replaying.

In recording mode all requests are made as they normally would be, but responses are saved, to be used in the replay mode.
In replay mode no requests reach the network. Any request that had a saved response receives that response, all others are left without receiving a response.

🚧

Request Matching

Requests must be matched with a previously made request to be replayed. Requests with the same host/path/method/query params/body are considered equal and can be used to replay. Changing values such as random ids or timestamps in network requests should be fixed to a constant when used with performance testing.

During the setup phase, replay is disabled, and network requests are processed as they usually would. A single iteration of the UI test is also run once with it disabled. A second iteration is run with network replay in the recording mode. All subsequent requests are made in the replaying mode. Note this means recording is only done for the base app. A request that is only found in the head app but not the base app won't be replayed.

Disk Restoration

To prevent the filesystem state from causing invalid performance test results, all files are restored to a fixed state at the start of a test iteration. Base and head apps receive the same filesystem state at the start of each test iteration. A snapshot of the filesystem (including the keychain for iOS apps) is saved before the first test iteration is run, for each iteration, the filesystem is restored to this state before the UI test runs.

[Android] CPU & GPU clock speeds

Android can control CPU and GPU clock speeds. Emerge locks these speeds to a preset value across all iterations of a performance analysis to ensure the OS can't dynamically change the CPU/GPU speed, which could significantly affect performance results.