Architecture

idb is formed of two components that have different responsibilities. Both these components are necessary for idb to run commands.

The idb cli

This is a python3 cli that exposes all of the functionality that idb has to offer. As it is written in Python, this does not need to be run from the Mac to which your iPhone or iOS Simulator is attached.

The cli itself is a thin wrapper for a client of the idb_companion. All communication to the idb_companion is done via gRPC. This can be either through TCP or a Unix Domain Socket.

This client library can be imported into your own python3 code if you wish, or the CLI can be called from any other kind of automation.

The idb_companion

The idb_companion is a gRPC server, written in Swift, that runs on macOS. It talks to the native APIs that are used for automating Simulators and Devices. It links the FBSimulatorControl and FBDeviceControl Frameworks, which are part of the overall idb project.

When the idb_companion acts as gRPC server, it does so for a single iOS target (a device or simulator).

A companion can outlive whatever started it, so it does not keep an inherited TMPDIR: it stages temporary files in the user's own temporary directory (getconf DARWIN_USER_TEMP_DIR), or in the directory named by the IDB_COMPANION_TMPDIR environment variable if that is set.

Additionally, the idb_companion has some commands that are deliberately unavailable from the python CLI, these operations are related to iOS Device management or operations on the lifecycle of a Simulator.

Connections

The idb cli will, by default operate in one of two modes:

• If the idb cli is running on macOS, then it will automatically start and stop companions for all targets that are attached to your Mac. This means that you can run commands against any iOS Simulators that you have, as well as any devices that you have connected.
• If the idb cli is running on any other OS, it will not manage companions for you. In this case you can either "attach" companions via idb connect or explicitly on every call using the IDB_COMPANION=hostname:port environment variable. This allows you to perform idb commands against companions running on other hosts. These facilities for companion discovery work on macOS also.

The transition to pure Swift

idb is transitioning to a pure Swift codebase. This is an ongoing migration, and the current state is:

• The idb_companion is Swift. The companion executable, its gRPC server and all of its request handling are implemented in Swift, replacing the previous Objective-C++ implementation that was built against the C++ gRPC library. The server is built on grpc-swift and SwiftNIO.
• The Frameworks are Swift. FBControlCore, FBSimulatorControl, FBDeviceControl and FBXCTestCore, including its DTX test-manager layer, are written in Swift. Objective-C remains only where Swift cannot express it: FBControlCoreObjC, a small library that catches exceptions thrown by Apple's private Objective-C API. Private frameworks such as CoreSimulator and MobileDevice.framework are reached through header-only declarations.
• New components are Swift-only. idb-repl and the companion's supporting libraries are pure Swift. idb-repl's whole dependency closure contains no Objective-C, which is what allows it to be built with the Swift Package Manager and cross-compiled for Linux.

For consumers of the Frameworks this means that their APIs are Swift protocols with async methods, as described in Swift Concurrency below.

Differences between Devices, Simulators & Emulators

iOS Devices and Simulators behave in substantially different ways, as well as Simulators behaving very differently to Emulators (for instance Android emulators):

• iOS Simulators and their child processes, appear as regular processes on the host operating system.
• iOS Simulators run executables that are native to macOS. This is unlike emulators for Android which may run across a variety of host operating systems, will always run native Android executables and may translate between ISAs.
• As iOS Simulators appear as native processes to macOS, many of the macOS level APIs for interacting with files and processes work just the same. This is useful in implementing Simulator functionality. The iOS Simulator uses the same kernel as the host macOS, which also goes some way to explain that some Xcode versions have increased macOS version requirements (there may be new kernel functionality in newer iOS versions that means that Simulators require this functionality through a macOS upgrade).
• iOS Simulators do not use exactly the same system frameworks as macOS. These Frameworks are implemented in the "Simulator Runtime" that is bundled within Xcode. The runtime contains Frameworks that are broadly the same as those on an iOS Device, except they are compiled for the macOS host architecture.
• As Simulators run natively, they have similar performance characteristics to that of the host. In a sense Simulator Applications perform in a similar way to a macOS application running on macOS. This usually means that Simulators are substantially more performant than emulators, even when an emulator has access to a hypervisor and is running the same ISA.
• iOS Simulators have the concept of a "root" directory, which can be thought of as the Simulator's Filesystem. Applications running inside the Simulator still have access to files outside of this root (there is no chroot'ing inside of the Simulator), so are able to manipulate files that are outside of this root. Applications are also able to manipulate files outside of their own "Application Sandbox" which is not the case on iOS Devices.
• This lack of isolation in iOS Simulators is a double edged sword. It can make certain automation cases more convenient to implement, but it is not easy to ensure that a Simulator has access to a limited amount of system resources. For example, emulators typically allow setting an upper limit on the number of cores or memory that can be consumed by the "guest" OS, where iOS Simulators can access the same resources that any application on the host can. This makes it harder to isolate multiple iOS Simulators running on the same host from each other. For instance, an iOS Simulator Application that consumes extreme amounts of system resources will exhaust these resources for other applications, processes or Simulators running on the same host.
• iOS Devices are very strict about isolating processes from each other as well as the host to which it is attached. Interaction between the host and attached iOS Device is only possible through purpose-built APIs that expose functionality for existing host-device behaviours. For instance, iOS App launching is implemented on iOS Devices through APIs that are used by the "Instruments" Application. This functionality is typically provided over a socket transport, with varying different protocol implementations depending on the domain. Access to these services is arbitrated via a lockdown service running on the iOS Device.
• As such, the implementations for functionality across iOS Simulators & Devices are drastically different, even within Xcode's own Frameworks.

Framework Concepts

There are two frameworks in FBSimulatorControl and FBDeviceControl that exist to implement the majority of the functionality used by idb. Additionally, there is the FBControlCore Framework that exists to define common interfaces for the Device and Simulator Frameworks and to provide other functionality that is common to both. Features that not every consumer needs, such as video, accessibility, test execution and code injection, are libraries of their own built on top: an FB…Core library shared by every kind of target, and an FBSimulator… library that adds the feature to Simulators. Installing is one too: FBArtifactStaging turns the archives and URLs that clients install from into bundles on disk, on top of FBControlCore alone. These Frameworks are able to be used independently of idb itself. This is an overview of how these Frameworks are designed together.

Each library has its own page: FBControlCore, FBVideoCore, FBArtifactStaging, FBXCTestCore, FBSimulatorBridgeProtocol, FBSimulatorControl, FBSimulatorVideo, FBSimulatorAX, FBSimulatorXCTest, FBSimulatorCodeInjection and FBDeviceControl.

Targets

An instance of a target (Target) is an object that represents a single iOS Simulator or Device. Target is a protocol definition that describes the functionality that is implemented by both Simulator and Device. This abstraction means that higher-level applications and Frameworks are able to treat a target the same, regardless of whether it is an iOS Simulator or Device.

As there are substantial differences in the way that iOS Simulators and Devices operate, this level of abstraction allows the Frameworks to smooth over the differences present in implementing common functionality.

Target Sets

A "Target Set" (TargetSet) represents a collection of targets. These are implemented in both SimulatorSet and DeviceSet. A Simulator set represents a root directory that is common to a number of Simulators. A Device Set represents all of the Devices attached to the host.

This abstraction allows for interfaces to "CRUD" operations on both Simulators and Devices, despite having different implementations. For instance the same API is used across Simulators and Devices for erasing them.

Configuration Values

Across the Frameworks, there are "Configuration" values. These are typically used for consolidating all the information required for a particular API call. For instance ApplicationLaunchConfiguration defines launch arguments, environment and launch modes.

These types exist so that APIs do not require extremely long and cumbersome argument lists, as well as providing sane defaults. These types are intentionally as behaviour-less as possible, close to pure value types.

Command Protocols & Implementations

In order to keep a common API between Simulators and Devices, idb has a set of protocols that define an interface for separate implementations across iOS Simulators & Devices. This does encourage the creation of well thought-out APIs. The idb_companion can be agnostic to the underlying iOS target, instead interacting with these protocols. The protocols are defined in Swift with async methods.

There may be some capabilities that are only supported by one or the other, depending on the target. For instance there is no concept of "Activation" on an iOS Simulator, so activate() is only declared on Device. Where a target does not support a given protocol, idb will fail when these APIs are called, giving an error message that explains what functionality is missing.

Implementations across Simulators & Devices are completely separated and implemented in their respective Frameworks. As an example, ApplicationCommands (which provides an API for launching and listing Applications on an iOS Target), is implemented separately in SimulatorApplicationCommands and DeviceApplicationCommands.

If functionality is common to both Simulators & Devices, its protocol is added to Target so that implementors of Target are required to implement it. For functionality that is not common, the relevant protocol is added to the definition of the concrete Simulator or Device class. For non-common protocols, the caller must either check for protocol conformance before calling an API, or use the concrete type directly.

Capabilities in libraries of their own

A feature that lives in its own library declares a capability protocol refining Target, such as VideoTarget in FBVideoCore or XCTestTarget in FBXCTestCore, and the library implementing it for a kind of target adds the conformance from outside that target's framework: FBSimulatorVideo adds extension Simulator: VideoTarget. FBControlCore never names the feature, and a binary that does not depend on the library does not link it.

The frameworks are compiled as whole modules, and Swift keeps every protocol conformance and type metadata record alive, so the linker can strip almost nothing from inside a framework. A library boundary is therefore the only way to keep a feature out of a binary. The flip side is that a conformance in a library is linked only if something references it: a caller holding any Target should name the concrete target types it handles rather than cast with as? any VideoTarget. The idb_companion does this once, when it chooses its target, and holds the result as a CompanionTarget that carries every capability its handlers use.

Logging

ControlCoreLogger is used throughout the codebase. This provides a common interface for logging out to system level loggers as well as files. Since all these Frameworks may be used in a variety of different scenarios, included where a logging client may be remote, this abstraction provides a common way of directing logs to the appropriate place. This is an "unstructured logger", which receives arbitrary strings.

Alongside it is a "structured logger" (EventReporter), which receives typed events rather than arbitrary strings. The idb_companion wraps every gRPC call in CompanionTelemetry, which reports one event per call, so the server produces accurate logging of all API calls that are made against it. EventReporter is a protocol, so a consumer can supply its own implementation and push these events into a datastore that supports aggregation.

IO

Due to the nature of the functionality that is offered in the Frameworks, IO is a very common task. There are a number of abstractions for reading and writing data between various sources and sinks. For example, the common interface (DescriptorReader) is used to read output from a spawned Application process and relay it to a consumer (FileWriter). This is then used to pipe Application output over idb's gRPC interface without the Application launcher having to be aware of what is consuming the output.

All of this is backed by libdispatch, due to its affordance for asynchronous IO, where file reading and writing is managed in an efficient manner without the user having to build their own IO multiplexer.

Swift Concurrency

A huge amount of the "work" that is done inside the Frameworks is based on IO and calling out to other APIs that perform IO. This work is very asynchronous, which means that there is a strong case for a consistent API for performing and waiting on this work.

That API is Swift Concurrency. Framework APIs are async and throws, so a failure carries a full error and a sequence of dependent calls reads as straight-line code. Where one task has to tell another that something has happened, such as a process reaching a state or a reader reaching the end of its input, FBControlCore provides AsyncEvent, which carries the first outcome to any number of waiting tasks. IO itself stays on libdispatch, as described above, and reaches callers as async calls.

idb_companion concepts

The majority of what the idb_companion does is to act as a gRPC server to all the functionality across the FBSimulatorControl and FBDeviceControl Frameworks. The server and all of its request handling are written in Swift.

It does have a handful of components that are important to the way that it operates.

main.swift

This is the entrypoint to the idb_companion and includes all of the various flags that are supported by it. This is used for specifying how the gRPC server should be started for a given iOS Target.

Additionally, it exposes a number of "CRUD" commands that are destructive for managing Simulators and Devices, for example --create, --erase and --delete. These commands are intentionally left out of the gRPC interface, to prevent unwanted behaviour. If you wish to perform destructive commands, these must be performed on the host where the iOS Simulators or Devices are present.

GRPCSwiftServer

This is the gRPC server itself, built on grpc-swift and SwiftNIO. It binds the idb.proto interface on a TCP port or Unix domain socket, optionally over TLS.

Method handlers

Each RPC in the idb.proto interface is implemented by its own Swift method handler, for instance InstallMethodHandler or AccessibilityInfoMethodHandler. A handler translates the request's protobuf types into the value types used by the Frameworks, awaits the underlying work with Swift Concurrency, and translates the result back into a protobuf response. This keeps each RPC's implementation small, isolated and testable.

IDBCommandExecutor

This provides a facade over the many APIs in the underlying Frameworks, so that the method handlers do not need to be aware of how they operate. It lives in CompanionLib, which also contains the storage management for installed artifacts.

Install telemetry

Every install call reports one event, and ends its completion log line with the same columns in braces, sorted by name after size: install succeeded in 2.10s {size=4096, extract_ms=30000, install_ms=40000, payload_kind=url, receive_ms=20000}. The column names and values below are stable. Dashboards and the install benchmark read them, so a change to any of them is a change to this contract. A column is absent when it doesn't apply, never zero.

Column	Type	Meaning
size	bytes	What reached the companion: streamed over the RPC for data, downloaded for url. For a failed download, what arrived before it failed.
payload_kind	data | url | file_path	How the client handed over what to install.
stream_format	zip | zip_zstd | tar_gzip | tar_zstd | tar | undetermined	The archive format, as its first bytes say, whatever the client declared.
extractor	zip_stream | zip_spool | tar_inprocess | bsdtar	The extractor that unpacked a streamed or downloaded archive.
zip_spool_fallback	0 | 1	For a zip, whether extracting it as it arrived failed and the spooled copy was extracted instead.
receive_ms	ms	Time to receive a streamed payload, or to download a URL.
ttfb_ms	ms	For a download, the time from the request to the response.
extract_ms	ms	Duration of the extract stage, which overlaps the download for a URL.
install_ms	ms	Duration of the install stage.
failure_stage	receive | download | extract | identify | install	The earliest stage still running when the call failed. receive means the streamed payload failed before any stage began. identify means what was staged held no artifact of the kind asked for.
failure_kind	http_status | transfer_failed | not_an_http_response | extraction_failed | no_installable_bundle | unidentifiable | a type name	What failed: an InstallError by its case, anything else by its Swift type.
cancel_source	client	Set when the call failed after the client cancelled the RPC.
ns_error_code, ns_underlying_error	domain:code	For a failed download, the error and the error beneath it.
cf_stream_error_code	int	For a failed download, the CFNetwork stream error, which names a TLS failure.
remote_address	host:port	For a failed download, the address the connection reached. Nothing from the URL is recorded, as its query carries signatures.
proxy_used	0 | 1	For a failed download, whether the request went through a proxy.
tls_trust_error, tls_chain	string, JSON	For a failed download whose server certificate was evaluated, why the host refused it and the chain it was shown.

The stages are those of an install's InstallProgressEvents: download (URLs only), extract (anything that is unpacked), and install. A streamed install reports progress to its client as InstallResponse.progress messages, but only for apps, because clients print every response to any other install as an installed artifact.