iOS Performance Testing

iOS performance testing runs your app on real, physical devices using Emerge's testing suite, allowing you to detect performance regressions for any PR before it's merged.

How it works

Each time a new version of the app is uploaded to Emerge, it gets uploaded to a real device in the cloud, along with the version it's being compared against (the app referenced from the specified baseSha). Then, for each version, a setup XCUITest is optionally run (e.g., logging the user in) before the UI test is run repeatedly that exercises the metric we want to test. Each run generates a single sample of that metric. Once we have a statistically significant result from all the samples, it gets shared via the /analysis endpoint alongside the app size result.

Integration

Marking metric start and end times

For each metric to be tracked, Emerge needs to know at what point it starts and stops. To do this, it uses NotificationCenter. To mark when a metric has started, use the following:

NotificationCenter.default.post(name: Notification.Name("EmergeMetricStarted"), object: nil, userInfo: [
    "metric": "myMetric",
])

To mark when a metric has finished, post the same notification but with "EmergeMetricEnded" as the name instead of "EmergeMetricStarted". Note that for measuring startup time, however, there's a special way to do it, because there's no way to post a Notification at the very start of a launch. To measure startup time, skip the "EmergeMetricStarted" notification, but add "use_process_start": true as a key-value pair alongside "metric": "myMetric". This will cause Emerge to use the process start time as the start of the metric.

Recording the metric

Setup

Some of your performance tests may need a setup XCUITest, that gets the app into a state where it can be efficiently tested repeatedly. For example, to test the app's startup time for when a user is logged in, you might have a setup XCUITest that logs the user in, and then the main test can just launch the app without having to log the user in/out.

If you need a setup XCUITest, create a .zip that includes a .xctestrun for it and email that to us.

Performance test

(Note: if you just want to test startup time for now, you can let us know, and don't need to upload anything following the steps below)

The actual performance test code is run from within Emerge's own XCUITest, where Emerge decides how many times and how exactly to run the test code to get more samples of the metric. So, instead of email a test bundle for this, email us a .framework that has the following class and method:

@objc class EMGTestRunner: NSObject {
    func runTest(app: XCUIApplication) {
        app.launch();
        // Further code here
    }
}

Emerge, in its wrapper UI test, will then call it repeatedly with something roughly equivalent to:

while shouldContinue {
    EMGTestRunner().runTest(app: app)
}