Coding with Titans

so breaking things happens constantly, but never on purpose

HowTo: Select running iOS simulator while automating testing

in  , , , ,

One thing was always a dilemma for me - running iOS automated unit-test from a command line. On the first look the command looks simple, it’s just a call to xcodebuild with a bunch of parameters. What can go wrong there, right?

$ xcodebuild -resultBundlePath "$test_results_path"
             -workspace "$workspace_path"
             -scheme "$scheme_name"
             -sdk "$sdk_version"
             -destination '$destination'
             -testPlan "$plan_name"
             -only-testing:'$single_test_path'
             test

Where:

  • test_results_path - describes the path, where to store test outcomes (.xcresult bundle).

  • workspace_path - points to the .xcworkspace location, which is gonna to be build. Alternatively we could use the path to .xcodeproj and rename the parameter from -workspace to -project.

  • scheme_name - describes the scheme from the workspace/project used for building.

  • sdk_version - can be either: iphoneos - for real devices or iphonesimulator in case a simulator is gonna be exercised. Exact version could be specified at the end of the current architecture selection: iphonesimulator14.2, however it’s not mandatory.

  • destination - defines the exact version of the simulator that will be used to launch all tests, example: "platform=iOS Simulator,name=iPhone 11,OS=13.3".

  • plan_name (optional) - the name of the actual .xctestplan that defined list of tests and additional arguments (like enabled/disabled code-coverage, randomization, zombie detection).

  • single_test_path (optional) - path to a file or function (without brackets!) inside the workspace to limit the number of executed tests, example: -only-testing:"CustomUITests/LoginTest/test_user_can_login".

Have you noticed, what is the biggest issue here?

The Problem!

Exactly - if we wanna run tests on the simulator we have to specify very detailed info about the model and iOS version running on it. And if you are a single developer without any dependencies or CI, it all seems fine. You always know, what machine will be used, what version of Xcode is installed, which simulators are there and probably all the tiniest details. Unfortunately things get a bit messy quickly as the build environment or team grows.

Take a look on a typical call:

$ xcodebuild -sdk iphonesimulator
             -workspace my_app.xcworkspace
             -scheme my_app
             -destination "platform=iOS Simulator,name=iPhone 11,OS=13.3"
             test

which is the source of pain. My related problems tend to be:

  1. Specifying the simulator like that might try to use not installed one as Xcode versions are updated every few months. All will obviously fail with older branches.

  2. Specifying the simulator version like that might might try to use the one not actively running at the moment. Instead it will spin the requested simulator and immediately destroy it after finishing test-plan execution. This can be a slow process on an old MacMini-2014 with only 4GB of RAM used as a CI machine.

  3. Adaptation to different environments is poor with own static “build script” with hardcoded simulator info and it gets tricky to run tests on multiple machines in parallel, while potentially utilizing different simulators and iOS versions using a single/simple command. Imagine each team-member’s machine, where everyone is forced to actually tweak the “build script” to match own environment.

Approaching the Solution

How do I want to solve all those issues then? Let’s try to make the simulator info a bit more dynamic! If we already have a “build script”, let’s try to extend it with some extra detection of simulators and their states. It simply can’t be a single hardcoded value anymore that fits all. It should rely more on the results of scanning the current machine and the custom logic we put there should inevitably take into account:

  1. requested simulator’s model (iPhone 8 or iPhone 11 Pro)

  2. requested system version (iOS 13.3 or iOS 14.2)

  3. should also check, what is currently running on the system (as could be more than a single simulator)

  4. check, what is available (installed) and could be potentially started-up

  5. making a sensible decision is the key part and what I present next might also require some tweaks for your setup!

So how to do it?

Prerequisites

Before we start we need some extra tooling. Beside Xcode, which is an obvious requirement, what is additionally extremely required for this HowTo to succeed is the jq utility. This little gem simplifies a lot JSON content processing from command line and makes all escaping and navigating over whole JSON tree structure sooo much more intuitive.

Here is a little tutorial on how to use it. But let’s make use of it.

My Solution

I must admit gathering all information about simulators on macOS was a non-obvious process. If you search for an ios all list simulators sentence on Google, it will suggest following command:

$ xcrun simctl list

It’s not quick at all and content of it is rather long and chaotic.

== Device Types ==
...
iPhone 11 (com.apple.CoreSimulator.SimDeviceType.iPhone-11)
iPhone 11 Pro (com.apple.CoreSimulator.SimDeviceType.iPhone-11-Pro)
...
== Runtimes ==
iOS 12.1 (12.1 - 16B91) - com.apple.CoreSimulator.SimRuntime.iOS-12-1
iOS 12.4 (12.4 - 16G73) - com.apple.CoreSimulator.SimRuntime.iOS-12-4
iOS 13.3 (13.3 - 17C45) - com.apple.CoreSimulator.SimRuntime.iOS-13-3
iOS 14.5 (14.5 - 18E182) - com.apple.CoreSimulator.SimRuntime.iOS-14-5
== Devices ==
-- iOS 12.1 --
    iPhone 8 Plus (AD96440F-70C5-44A6-AFAF-0002643F0653) (Shutdown) 
    iPhone X (5A75B96C-BAC6-43BB-854B-FB79CA3B4CAC) (Shutdown) 
...
-- iOS 13.3 --
    iPhone 11 (6F36FE2B-B359-4828-85BF-48FA8725BD42) (Shutdown) 
    iPhone 11 Pro (6DE6FB5D-5163-4864-90F7-DFDF72B12F67) (Shutdown) 
    iPhone 11 Pro Max (F5EAF268-E6A0-4964-9066-B0038158A872) (Shutdown) 
...
-- iOS 14.5 --
    iPhone 8 (AA4A669C-CB4D-4B6F-8049-6970FAFEA95D) (Shutdown) 
    iPhone 8 Plus (62AB17EE-A7D3-4179-9C63-A102023F1D74) (Shutdown) 
    iPhone 11 (ECB2E757-C518-4510-B23C-899160956CD6) (Shutdown) 
    iPhone 11 Pro (565F05AA-086B-4232-BC26-8151456A77B0) (Shutdown) 
    iPhone 11 Pro Max (D64F9B8C-1D38-443B-8939-E46BD0784349) (Shutdown) 
...
-- Unavailable: com.apple.CoreSimulator.SimRuntime.iOS-13-6 --
    iPhone 8 (149859D8-2E56-4DAC-B6EB-4517980C16FC) (Shutdown) (unavailable, runtime profile not found)
    iPhone 8 Plus (0C9DB6C3-07E8-4BD6-B5E4-F20B8E5E4CB7) (Shutdown) (unavailable, runtime profile not found)
    iPhone 11 (2EBD518D-4934-4148-8049-4FD237DFD3E9) (Shutdown) (unavailable, runtime profile not found)
...
== Device Pairs ==
D7CDA4CA-51EE-4B9E-B9A6-4FB6B3855205 (active, disconnected)
    Watch: Apple Watch Series 5 - 40mm (A3874C82-DC56-4BD4-9A04-DEE53F3494B4) (Shutdown)
    Phone: iPhone 12 mini (BE5B7D72-4F0E-4BDE-B160-57F8EA7FA1D5) (Shutdown)
FC236898-DA0C-4E08-9357-4FD7B5FE75EB (active, disconnected)
    Watch: Apple Watch Series 6 - 40mm (5FC5D68B-7EC2-4347-99B7-93685629E749) (Shutdown)
    Phone: iPhone 12 Pro (36372449-AA50-44CF-9531-5F4E8C9F750D) (Shutdown)
...

Not very useful at the moment, plus data extraction of this format would require some strange parser, right? Well, if you play a bit with other arguments that can be passed to simctl, you might notice, its possible to change the output to JSON. Let’s fire a command:

$ xcrun simctl list --json devices available

Producing something more developer-friendly:

{
  "devices" : {
    "com.apple.CoreSimulator.SimRuntime.tvOS-13-2" : [
    ],
    "com.apple.CoreSimulator.SimRuntime.watchOS-6-2" : [
    ],
    "com.apple.CoreSimulator.SimRuntime.iOS-13-4" : [
    ],
    ...
    "com.apple.CoreSimulator.SimRuntime.iOS-13-3" : [
      {
        "dataPath" : "\/Users\/pawel\/Library\/Developer\/CoreSimulator\/Devices\/2B174A3C-078A-4C93-861B-CF7D733FEA8A\/data",
        "logPath" : "\/Users\/pawel\/Library\/Logs\/CoreSimulator\/2B174A3C-078A-4C93-861B-CF7D733FEA8A",
        "udid" : "2B174A3C-078A-4C93-861B-CF7D733FEA8A",
        "isAvailable" : true,
        "deviceTypeIdentifier" : "com.apple.CoreSimulator.SimDeviceType.iPhone-8",
        "state" : "Shutdown",
        "name" : "iPhone 8"
      },
      ...
    ],
    "com.apple.CoreSimulator.SimRuntime.iOS-14-5" : [
      {
        "dataPath" : "\/Users\/pawel\/Library\/Developer\/CoreSimulator\/Devices\/AA4A669C-CB4D-4B6F-8049-6970FAFEA95D\/data",
        "logPath" : "\/Users\/pawel\/Library\/Logs\/CoreSimulator\/AA4A669C-CB4D-4B6F-8049-6970FAFEA95D",
        "udid" : "AA4A669C-CB4D-4B6F-8049-6970FAFEA95D",
        "isAvailable" : true,
        "deviceTypeIdentifier" : "com.apple.CoreSimulator.SimDeviceType.iPhone-8",
        "state" : "Shutdown",
        "name" : "iPhone 8"
      },
    ...
    {
        "dataPath" : "\/Users\/pawel\/Library\/Developer\/CoreSimulator\/Devices\/ECB2E757-C518-4510-B23C-899160956CD6\/data",
        "logPath" : "\/Users\/pawel\/Library\/Logs\/CoreSimulator\/ECB2E757-C518-4510-B23C-899160956CD6",
        "udid" : "ECB2E757-C518-4510-B23C-899160956CD6",
        "isAvailable" : true,
        "deviceTypeIdentifier" : "com.apple.CoreSimulator.SimDeviceType.iPhone-11",
        "state" : "Booted",
        "name" : "iPhone 11"
      },
      {
        "dataPath" : "\/Users\/pawel\/Library\/Developer\/CoreSimulator\/Devices\/565F05AA-086B-4232-BC26-8151456A77B0\/data",
        "logPath" : "\/Users\/pawel\/Library\/Logs\/CoreSimulator\/565F05AA-086B-4232-BC26-8151456A77B0",
        "udid" : "565F05AA-086B-4232-BC26-8151456A77B0",
        "isAvailable" : true,
        "deviceTypeIdentifier" : "com.apple.CoreSimulator.SimDeviceType.iPhone-11-Pro",
        "state" : "Shutdown",
        "name" : "iPhone 11 Pro"
      },
      ...
    ]
  }
}

And now we are almost there. Thanks to this question from StackOverflow.com - we know, how to process this output to get a streamlined info of only valid installed simulators with an iOS version attached into its name:

# copied Jeff's suggestion:
$ xcrun simctl list --json devices available | jq '.devices | to_entries[] | (.key | capture("com\\.apple\\.CoreSimulator\\.SimRuntime\\.iOS-(?<version>.+)")) as {$version} | .value[] | {name: "\(.name) (\($version|sub("-"; ".")))", udid}'

Printing:

{
  "name": "iPhone 8 (13.3)",
  "udid": "2B174A3C-078A-4C93-861B-CF7D733FEA8A"
}
{
  "name": "iPhone 8 Plus (13.3)",
  "udid": "1B0A1DFD-47E8-459F-8EDE-50BAE8A902C3"
}
{
  "name": "iPhone 11 (13.3)",
  "udid": "6F36FE2B-B359-4828-85BF-48FA8725BD42"
}
{
  "name": "iPhone 11 Pro (13.3)",
  "udid": "6DE6FB5D-5163-4864-90F7-DFDF72B12F67"
}
{
  "name": "iPhone 11 Pro Max (13.3)",
  "udid": "F5EAF268-E6A0-4964-9066-B0038158A872"
}
...

Unfortunately, the outcome it produces is not a valid JSON array and it can be improved even more by adding iOS version as separate attribute, along with the current state (lowercase!) of the simulator. Let’s welcome my ultimate query to get the list:

$ xcrun simctl list --json devices available | jq '.devices | [ to_entries[] | (.key | capture("com\\.apple\\.CoreSimulator\\.SimRuntime\\.iOS-(?<version>.+)")) as {$version} | .value[] | {name: "\(.name)", version: "\($version|sub("-"; "."))", udid, state: "\(.state|ascii_downcase)", dataPath} ]'

Printing:

[
  {
    "name": "iPhone 8",
    "version": "13.3",
    "udid": "2B174A3C-078A-4C93-861B-CF7D733FEA8A",
    "state": "shutdown",
    "dataPath": "/Users/pawel/Library/Developer/CoreSimulator/Devices/2B174A3C-078A-4C93-861B-CF7D733FEA8A/data"
  },
  {
    "name": "iPhone 8 Plus",
    "version": "13.3",
    "udid": "1B0A1DFD-47E8-459F-8EDE-50BAE8A902C3",
    "state": "shutdown",
    "dataPath": "/Users/pawel/Library/Developer/CoreSimulator/Devices/1B0A1DFD-47E8-459F-8EDE-50BAE8A902C3/data"
  },
  {
    "name": "iPhone 11",
    "version": "13.3",
    "udid": "6F36FE2B-B359-4828-85BF-48FA8725BD42",
    "state": "booted",
    "dataPath": "/Users/pawel/Library/Developer/CoreSimulator/Devices/6F36FE2B-B359-4828-85BF-48FA8725BD42/data"
  },
  {
    "name": "iPhone 11 Pro",
    "version": "13.3",
    "udid": "6DE6FB5D-5163-4864-90F7-DFDF72B12F67",
    "state": "shutdown",
    "dataPath": "/Users/pawel/Library/Developer/CoreSimulator/Devices/6DE6FB5D-5163-4864-90F7-DFDF72B12F67/data"
  },
  ...

Now, with this information , we can finally apply some logic above it. Let’s move into bash script, where we can express it as set of functions to actually produce requested content:

  1. Create a script file (simulator.sh) with a header:

    #!/bin/sh

  2. Add a function to list simulators as defined above:

    list_simulators() {
    local list_json=$(xcrun simctl list --json devices available | jq '.devices | [ to_entries[] | (.key | capture("com\\.apple\\.CoreSimulator\\.SimRuntime\\.iOS-(?<version>.+)")) as {$version} | .value[] | {name: "\(.name)", version: "\($version|sub("-"; "."))", udid, state: "\(.state|ascii_downcase)"} ]')
    echo "$list_json"
}

  3. Add a function that filters the list of simulators (given in JSON) by state, name and system version:

    filter_simulators() {
    local list_sim_json="$1"
    local filter_state="$2"
    local filter_name="$3"
    local filter_version="$4"

    if [ "$list_sim_json" == "" ]; then
        list_sim_json=$(list_simulators)
    fi

    local arg=".[]"
    if [[ "$filter_state" != "" && "$filter_state" != "any" ]]; then
        arg="$arg | select( .state == \"$filter_state\" )"
    fi
    if [ "$filter_name" != "" ]; then
        arg="$arg | select( .name | startswith(\"$filter_name\") )"
    fi
    if [ "$filter_version" != "" ]; then
        arg="$arg | select( .version | startswith(\"$filter_version\") )"
    fi

    if [ "$arg" != ".[]" ]; then
        list_sim_json=$(echo "$list_sim_json" | jq "[ $arg ]")
    fi
    echo "$list_sim_json"
}

  4. Finally add a function that applies a logic, to first find any running simulator with expected name and version, then prints its spec in desired format.

    select_simulator_spec() {
    local filter_name="${1:-iPhone 11}"
    local filter_version="${2:-13.}"

    # is the filtering already matching the spec?
    # if so, return to use it
    if [[ $filter_name == platform* ]]; then
        echo "$filter_name"
        return
    fi

    # take a running device
    local simulators=$(list_simulators)
    local devices=$(filter_simulators "$simulators" "booted" "$filter_name" "$filter_version")

    if [[ "$devices" == "[]" || "$devices" == "" ]]; then
        # or any other booted
        devices=$(filter_simulators "$simulators" "booted")
    fi

    if [[ "$devices" == "[]" || "$devices" == "" ]]; then
        # or any other available on the machine
        devices=$(filter_simulators "$simulators" "" "$filter_name" "$filter_version")
    fi

    # or any other with incremented version...
    if [[ "$devices" == "[]" || "$devices" == "" ]]; then
        devices=$(filter_simulators "$simulators" "" "$filter_name" "13.")
    fi
    if [[ "$devices" == "[]" || "$devices" == "" ]]; then
        devices=$(filter_simulators "$simulators" "" "$filter_name" "14.")
    fi
    if [[ "$devices" == "[]" || "$devices" == "" ]]; then
        devices=$(filter_simulators "$simulators" "" "$filter_name" "15.")
    fi

    # format the result
    if [[ "$devices" == "[]" || "$devices" == "" ]]; then
        echo "/fail/"
        exit 1
    else
        # expectation: "platform=iOS Simulator,name=iPhone 11,OS=13.3"
        local result=$(echo "$devices" | jq -r '.[0] | "platform=iOS Simulator,name=\(.name),OS=\(.version)" ')
        echo "$result"
    fi
}

    and run this function by stating at the end of the file:

    select_simulator_spec "$1" "$2"

    And we are done.

If you created a single script out of the all pieces above, called simulator.sh, then you can execute it to find proper spec valid for your machine as:

$ ./simulator.sh "iPhone 11 Pro" "13.3"

# or simply:

$ ./simulator.sh "iPhone 11 Pro"

To receive, when no other simulator was booted yet:

platform=iOS Simulator,name=iPhone 11 Pro,OS=13.3

or, when iPhone 11 with iOS 13.3 was running in the background:

platform=iOS Simulator,name=iPhone 11,OS=13.3

Other useful commands

You can find more useful simulator commands here.

Most often used one of them is how to start a selected set of simulators:

# select simulators to boot-up:
# iPhone 11, iOS 13.3
$ xcrun simctl boot 6F36FE2B-B359-4828-85BF-48FA8725BD42

# iPhone 11, iOS 14.5
$ xcrun simctl boot ECB2E757-C518-4510-B23C-899160956CD6

# launch the simulator app:
$ open /Applications/Xcode_12.5.app/Contents/Developer/Applications/Simulator.app

Summary

As I tried to show in this post. There is a way to list existing simulators, along with the possibility of filtering only “booted” ones. The conditions I applied to favor running simulators over the expected exact compliance with requested name & version might not be totally valid for you. Please experiment and tune it as you like. Make the whole build and tests execution a smooth process.

Hopefully, thanks to the presented solution above, testing won’t turn red anymore based only on some environmental configuration differences.

Greetings!