Overview
Interested in using Emerge's Performance Analysis suite?
Performance Analysis is a new product that needs to be enabled for your account. Get in touch with our team to enable for your organization.
Performance Analysis allows you to detect regressions and improvements in any of your key performance metrics, such as startup time, for a PR before it's merged.
For each test, two copies of the app are uploaded to a real, physical phone in the cloud (typically the SHA of a PR and the base SHA it's branched off of). The phone runs performance tests for each copy until it comes to a statistically sound conclusion about each metric and if it has regressed, improved, or remained the same.
Variance Control
Both iOS and Android performance analyses are designed to give you accurate results of the performance impact of a change. This is done by running a test repeatedly, with and without the change you're measuring, until there are enough samples to give a statistically significant result. Ideally, the code executed in each test would be exactly the same, but there are many sources of variance that can cause different code paths to be taken in each test run. For example, you might have some code that generates a random number and uses the result to decide what to do. This kind of code won't work well with performance testing, since it will perform differently over multiple runs even for one build of the app.
To ensure performance testing works with most apps, many sources of variances are automatically controlled by Emerge. For example, any network request made by the app is automatically cached and replayed, so a given request always produces the same response. The disk state is also reset between tests, so if the app writes to a file during a test, that file is restored to its initial state before the test runs again.
As a general rule of thumb, performance testing aims to control everything about the app's environment, so the only changes measured will be changes from within the uploaded app. This means a single upload will always take the same code paths while running in performance analysis.
Suppose you're trying to test something that is outside of the app's control, such as the performance two different users would see for the same app. In that case, the additional user data should be included in two separate app uploads (such as in a configuration plist on iOS or with BuildConfig settings on Android). These two different uploads can then be compared.
Network Replay
To prevent different network responses from causing invalid performance test results, all HTTP/HTTPS requests are intercepted and controlled. Base and head apps receive the same network state for each test iteration. Network Replay operates in three modes - disabled, recording, and replaying.
In recording mode all requests are made as they normally would be, but responses are saved, to be used in the replay mode.
In replay mode no requests reach the network. Any request that had a saved response receives that response, all others are left without receiving a response.
Request Matching
Requests must be matched with a previously made request to be replayed. Requests with the same host/path/method/query params/body are considered equal and can be used to replay. Changing values such as random ids or timestamps in network requests should be fixed to a constant when used with performance testing.
During the setup phase, replay is disabled, and network requests are processed as they usually would. A single iteration of the UI test is also run once with it disabled. A second iteration is run with network replay in the recording mode. All subsequent requests are made in the replaying mode. Note this means recording is only done for the base app. A request that is only found in the head app but not the base app won't be replayed.
Disk Restoration
To prevent the filesystem state from causing invalid performance test results, all files are restored to a fixed state at the start of a test iteration. Base and head apps receive the same filesystem state at the start of each test iteration. A snapshot of the filesystem (including the keychain for iOS apps) is saved before the first test iteration is run, for each iteration, the filesystem is restored to this state before the UI test runs.
FAQ
iOS Performance Analysis
Emerge supports testing the performance of any iOS flow using XCUITests.
Read the full iOS Performance Analysis documentation.
Android Performance Analysis
Emerge supports testing the performance of any Android flow using UIAutomator tests.
Updated 7 months ago