[iOS] How Carthage Works: Migrate from CocoaPods to Carthage to Save Xcode Clean Build Time (Legacy)
Revision: After suffering, I don’t think right now that it is a good idea to use Carthage. Carthage is too vulnerable to Xcode/Swift updates and it became a disaster every time we failed to compile the app just because we upgraded Xcode or even libraries. It’d be better to go back to CocoaPods or move forward to SPM as they are source code oriented and so much less prone to such changes.
But I regard this article as still valuable as it was my journey to explore how Xcode deals with modules. The following was written when I still believed Carthage could be one of my options to maintain modules. I guess it might be still worth your time reading it hopefully to gain benefits :)
It recently took more and more time to build my company’s app in about lengthy 7 minutes or even more. This was really a bother when we switch between git branches, causing Xcode to perform almost clean build. We began to explore some way to reduce the build time of our app. One solution we started with is to migrate some of the frameworks from CocoaPods to Carthage, a recently popular tool to prebuild dependencies. After 2 months of trial and errors, we successfully reduced the clean build time to about 5 minutes, saving 20% of our precious time.
In this article, I’ll cover what I learned during the migration, hoping that other developers can benefit. Here is the outline:
- Why CocoaPods makes building slow
- What is Carthage
- A simple migration — to bootstrap dependencies
- A difficult migration — to apply patches during bootstrapping
- Share our prebuilt frameworks with other developers
- Lots of errors we may encounter
- Summary
1. Why CocoaPods makes building slow
Our app used CocoaPods, a widely used tool because of its easiness, to manage dependencies. CocoaPods dynamically generates Xcode projects for the dependencies based on the pods listed in Podfile. Every clean build, therefore, leads to the frameworks completely rebuilt from the generated projects from scratch. Even on incremental build, Xcode also spends some time checking for the changes to the generated projects and sometimes triggers rebuild.
The upside of CocoaPods is that it always compiles the frameworks along with our app so there will be no issues like incompatible Swift version. The downside, on the other hand, is to make us suffer from unnecessarily rebuilding the frameworks that usually don’t change at all. If we can prebuild the frameworks in advance, plus we have confirmed that the frameworks are compatible to our app, then Xcode needs to clean build only our app itself.
Nonetheless, it was really a painful process to migrate a dependency from CocoaPods to Carthage because I didn’t have enough knowledge before I began. I decided to share lessons that Carthage taught me so others don’t need to suffer as I did :P.
2. What is Carthage?
Carthage is a tool that facilitates the building of dependencies/frameworks. And that’s it. It doesn’t link the frameworks for us. Carthage, by design, is responsible only for building them. All things other than building relies on ourselves. We need to link them manually to our app on our own.
The dependencies of our app, along with our required version constraints, are listed in Cartfile, a special text file for Carthage to read. Before building the frameworks, Carthage needs to resolve their exact versions compatible to the constraints in our Cartfile. The resolved version will be saved in another text file — Cartfile.resolved, which is the most important file for later building process. For every dependency in Cartfile.resolved, the building process is bootstrapped in two steps:
- Checkout: Clone the GitHub repository of the resolved version, decided by Cartfile.resolved, into our local path. Recursively clone more repositories if the dependency has its own Cartfile to indicate its dependencies too, until all required repositories for the dependency to work are cloned locally.
- Build: Dive into our local repositories we just cloned. Build the product for each of the shared/public schemes in the Xcode project of each repository. Carthage will dive into and build a dependency’s child dependencies recursively before building it. If we don’t specify the configuration, Carthage defaults to build with Release configuration.
The two main steps Carthage takes sound simple. Without Carthage, we need to pull the repositories of our dependencies by ourselves. We need to build them with their Xcode projects and check if we need to pull to build more frameworks if our dependencies also depend on other frameworks. Carthage do save such work for us. Note that Carthage improves mainly the clean build time. The saved time in incremental build, however, is very limited.
We have learnt how Carthage works. Let’s move on to practice!
3. A Simple Migration
Suppose we have an app with dependencies managed by CocoaPods. The very first step before migration is to check if the dependency has Carthage support. That usually means: (a) it has an Xcode project on Github for us to checkout and build, and (b) if it relies on other dependencies it also provides a Cartfile to refer to them.
Migrate frameworks from Podfile to Cartfile
If the dependency does have Carthage support, we can start migration. For example, let’s say we have a Podfile like:
pod "PromiseKit", "~> 6.0"Note that the line in Podfile refers to a registered pod in the CocoaPods repository. CocoaPods needs its podspec to generate its target. While the line in Cartfile refers to a public Github repository, which Carthage can clone and checkout in order to pre-build the framework product.
Suppose we have found its Carthage-supported repository on GitHub: https://github.com/mxcl/PromiseKit. We can remove the one from Podfile and then add the line to Cartfile:
github "mxcl/PromiseKit" ~> 6.0Build dependencies with Cartfile.resolved
Carthage builds dependencies in two steps: checkout and build. We can bootstrap them in one simple command:
carthage bootstrap --platform iosThe bootstrapping process relies on Cartfile.resolved (not Cartfile!) to know which framework version to build. For the first time when we run bootstrapping, this file is not present, so Carthage will create one for us — meaning Carthage will resolve the dependency versions according to the constraints listed in Cartfile. For example, the constraint ~> 6.0 in Cartfile may resolve the latest compatible version 6.8 in Cartfile.resolved. After we have Cartfile.resolved, everyone who has the resolved file will build the same version of the dependencies as ours when he runs carthage bootstrap --platform ios.
The flag —-platform ios is optional. Carthage builds all shared schemes in the Xcode project by default, and the flag is currently the only filter that allows us to select our wanted schemes to build based on the supported platform of its target.
How to update Cartfile.resolved: The resolved file is fixed at first. In the future, if we want to add/remove dependencies or upgrade the versions to the latest compatible ones, we need to edit Cartfile and run the command carthage update --platform ios —-no-checkout to update Cartfile.resolved. Discard the flag --no-checkout to update followed by bootstrapping.
Link and copy frameworks to the app
After building process is done, we should have the pre-built frameworks ready! Suppose we have an app with the app delegate:
To make this code compiled, we need to (a) make the imported framework searchable by the compiler, (b) link the framework to the app target so the symbols can be connected, and (c) add a copy-framework build phase to copy frameworks into the app bundle if they are built as dynamic ones.
$(SRCROOT)/Carthage/Build/iOS/XXX.framework` and output paths `$(BUILD_PRODUCT_DIR)/$(FRAMEWORK_FOLDER_PATH)/XXX.framework`. Note that we don’t need to do this if they are static frameworks.It’s worth to mention the required Copy Frameworks build phase suggested by Carthage. What it does is to copy things from the input paths to the output paths. It’s almost the same as the build-in Embed Frameworks build phase, except that Copy Frameworks does more necessary works such as copying dSYMs generated by Carthage. So, a framework should not be proceeded by the two phases at the same time. Feed it only to Copy Frameworks when it is built by Carthage.
Moreover, this step is only for dynamic frameworks because they need to be packaged into the app bundle for the app to load at run time. We should skip this step for static frameworks because static ones have been compiled into the app at build time. Despite that, we don’t need to worry for now because most of the Carthage-supported frameworks are dynamic if we don’t touch their build settings. We’ll talk about this topic later.
Finally, compile and run. Congrats we have done migration ;)
We might not need every generated framework
Notice that running bootstrapping of a dependency could generate multiple frameworks because it may depend on other frameworks. In this case we need to link all of them to our app to make it work. To give an example, suppose that Cartfile of Framework A says that it depends on Framework B. If my app depends on Framework A, bootstrapping Framework A with Carthage will bring the bootstrapping of Framework B. To make Framework A work, I need to link both Framework A and Framework B to my app.
On the other hand, sometimes it is also possible that we don’t need to link some of the generated frameworks which come from the shared/public schemes that we don’t need. For instance, our app depends on Framework X, but the Xcode project of Framework X contains the shared scheme for Framework X and the other for Framework Y, an optional framework provided by this Xcode project. Carthage will build both from the Xcode project but we only need to link Framework X. That is, Carthage will build products according to all shared schemes in the Xcode project, so be careful to decide which generated frameworks to link to our app.
But, in the previous example, if we want Framework X for iOS while we found Framework Y with the supported platform for MacOS — we can filter the schemes with —-platform ios so that Carthage won’t build Framework Y. This filter is the only way we can avoid unneeded schemes. However, if Framework Y supports iOS as well, there is no other way in a single bootstrapping of an Xcode project to easily deselect the scheme for Framework Y.
4. A Difficult Migration
Sometimes we need some more custom configuration on the dependencies. In Podfile, it is possible to achieve that using pre_install and post_install hooks. One of my use cases is to make the frameworks built as static ones to speed up the app launch time. Generally speaking, Carthage builds dynamic frameworks. However, that’s not decided by Carthage, but by the dependencies themselves!
To make the dependency built as static ones, we need to go to our cloned local repository in the path Carthage/Checkouts, open its Xcode project’s build settings, and set the Mach-O Type from Dynamic Library to Static Library. That is, we have to modify the Xcode project. If now we click on the build button in Xcode, it starts to build static framework products.
Apply patches to the Xcode project of a dependency
So, the question is: how do we achieve that with Carthage? How can we make changes to the build settings before Carthage builds the products? We proposed two options: (a) fork the repository with the changes or (b) dynamically apply patches during the bootstrapping.
Fork the repository with changes: One principle of Carthage is to leave all the settings to the dependency itself. Carthage only checks out and builds it, and nothing more. It’s assumed the provider’s duty to prepare the right build settings. Following this principle, we should fork the repository to make necessary patches, and reference Cartfile to our forked one.
The downside of this solution is the maintenance cost. There is a discussion over this — we need to fork every framework that we need to be static. It’s also a pain to keep our forked ones up-to-date with the original repositories. Hence, another option comes out.
Dynamically apply patches during bootstrapping: We know Carthage bootstrapping consists of two steps: checkout and build. We did it in one command carthage bootstrap, but we can also call individual carthage checkout followed by carthage build. The idea is: we can patch changes to the Xcode project’s build settings after we checkout the repository and before we build the frameworks.
When it comes to the solution, we can’t call Carthage’s provided bootstrap command anymore. A custom script is therefore unavoidable, as discussed here and here. We need to, in turn, call (a) carthage checkout [dep], (b) apply patches to it, and (c) call carthage build [dep]. I made a Podfile-like script here, which is used to replace Cartfile.
This way, we avoid forking every repository just for little customization. Nonetheless, this breaks the principle that we should never do anything other than simply checkout and build with Carthage, making the bootstrapping dirty and unstable.
Either of the two ways to patch the projects has their pros and cons. Choose the one we prefer, even though both of them do increase the cost of maintenance. It is the trade-off if what we want differs from what the dependency project provides.
Recap one important thing we just mentioned: remember to remove the static frameworks from the Copy Frameworks build phase. Of course, we may not only want to build static frameworks, but also want to build them with dSYMs, make them Swift module stable, distribute them with bitcode, and even remove unwanted shared schemes — We can achieve all these by applying patches.
5. Share Our Prebuilt Frameworks with Other Developers
This is an optional topic. Suppose we are working on a collaborative team. We have to distribute Cartfile.resolved to other developers and ask each of them to run carthage bootstrap —-platform ios so as to prebuild their local frameworks before they start development. Make sense, huh?
It’s time to face Carthage’s known slowness. If other developers are impatient with slow prebuild time with Carthage, maybe instead of Cartfile.resolved, we can try distributing the frameworks that we just prebuilt to others. This way only we, the dependency maintainer, need to prebuild the frameworks. Others just do nothing more than download the prebuilt products. We have some solutions to achieve this:
- A shared cache system for everyone to access and update. An existing solution like Rome utilizes AWS to attain framework sharing. The downside is that everyone would share the same cache so no on is allowed to be out of date.
- Git LFS also lets us share our frameworks with others by adding them in the git repository. The built framework is under version control so everyone is allowed to have the ones corresponding to his commit. This is the solution that was adopted in my case. The repository is required to support Git LFS.
Either way, Cartfile and Cartfile.resolved becomes useless to other developers except for the dependency updater. Everytime the updater wants to add/remove/upgrade dependencies, he needs to maintain Cartfile and Cartfile.resolved to manage which ones he is going to build or rebuild before sharing the updated products to others.
6. Lots of Errors We Possibly Encounter
It seems simple to go through the concepts, but it requires quite a lot of time to go through trial and error. More issues may pop up along the migration. Let’s recap the steps that compose the framework migration, and then share some situations we experienced by three categories:
- Compile-time: A framework are pre-compiled by Carthage. And we link it to our app. The app imports the framework to use it. Finally, the app should be compiled with the imported framework. If the framework is static, it is compiled into the app binary.
- Runtime: Our app launches. If the framework is dynamic, the app will load it. The app crashes if it fails to find the dynamic framework in the app bundle.
- Archive: We package the app with necessary binaries and assets into the archive for distribution such as uploading it to App Store. Recompilation with Bitcode could be necessary so frameworks must also provide Bitcode.
Compile-time errors
No such module: This happens when we fail to compile the app. Linking the framework to the app does not mean the app can find our framework. Check if the framework search path in our build setting to see if our framework exists in the paths.
When we run the app, the compiler does three things: (a) search the frameworks in the search path to compile, after that (b) link the object files of the frameworks to our app, and (c) load the dynamic frameworks when the app is running. Identifying which phase to emit the error could help us find out the cause.
Undefined symbols for architecture x86)64: ___llvm_profile_runtime
ld: symbol(s) not found for architecture x86_64: The code coverage in our framework probably goes different from our app’s code coverage. Needs rebuilding.
Runtime errors
dyld: Library not loaded: The dynamic framework is not found on runtime. The most possible reason is we haven’t build the dynamic frameworks in the Carthage directory. While another possible situation is that we might build it as static but we also have a dynamic one in the search path, so the app mistakes the dependency as dynamic one and finds it not copied in the bundle.
Archiving / Distribution errors
Firebase.framework does not contain bitcode: If we’re building Firebase, some of the downloaded frameworks acts only the bridge in compile time. Existing in the search path is their only requirement. Don’t link them to the app or those which do not contain bitcode cannot be linked on archive time. More info here.
ld framework not found: If we archive our app with this error, we probably have not add our dynamic frameworks in the copy framework phase.
exportArchive: IPA processing failed: This happens when we successfully archive our app but failed to export as an .ipa file such as ad-hoc distribution. It’s likely that we copy and embed the frameworks at the same time. Please remove the embed one.
XXX is implemented in both YYY and ZZZ . One of the two will be used. Which one is undefined: The XXX symbol is duplicated in the runtime. Check if we copied a static framework that causes this issue.
Ld xxxxx normal arm64: Failure when exporting archive. Probably we have incompatible bitcode. Bitcode is a intermediate code that is required when distributing it to Apple to recompile our app. Please ensure all our dependencies have the bitcode option ON in the build settings. The way to check if a built framework contains bitcode:
otool -l XXX.framework/XXX | grep __LLVMCarthage also downloads binaries if available. If the prebuilt ones have no bitcode, build it with —-no-use-binaries to avoid that.
exportArchive: exportOptionsPlist error for key ‘method’:
expected one of {} but found ad-hoc: After archiving, only app bundle can be exported. If more than one binary is included in the archive bundle directory, it will be not treated an app bundle so no distribution is allowed. We may need skipping install in the build setting, or check if there are more than one project in our code base.
7. Summary
It is not easy to explore Carthage migration but it pays once we discover that the clean build time dramatically decreases and we also learned how Xcode imports dependencies which we may be not familiar with before. Please also share your experience if you’ve also gone through the process!
If this article does help you, please give me a applause to let me know ☺