Scaling Build and Merge Times for iOS and Android
Native mobile app build times become an issue very quickly. iOS engineers are very familiar with how seemingly slow Xcode builds projects, and Android engineers with a project beyond “Hello, World” will also start to count the seconds - then the minutes on a build.
Neither Apple, nor Google have prioritized build time improvements for large projects. This is especially true for large projects, with hundreds of thousands, or millions of lines of code, and dozens of dependencies.
Luckily, there are tools to make build times faster - but they take time to integrate, to tweak and to fit into the “one-click-build” workflow. Bazel is the tool that is gaining the most popularity among companies building mobile at scale - many of them switching from Buck or Gradle, Uber, Pinterest and even the Android AOSP platform are switching over. There are many improvements that can be made under the hood. The more mobile engineers in a company, the more it can make sense to have engineers work part-time or full-time on improving the app build experience.
Focusing on build improvements becomes a must, on a per-platform level, as the mobile developer team grows. As an example, on iOS, dynamic frameworks can be costly both for build times and for app size. When shorter build times and smaller app size is a priority, consider switching to static libraries.
Distributed source code vs a monorepo is a question that teams at the size of 20-40 engineers start to question. Most teams start with pulling in dependencies and libraries from different repositories, doing this before compile time. The time to download dependencies through the network is time-consuming, so caching is used to optimize this approach.
As the codebase and the number of engineers working on the codebase grows, the time spent fetching dependencies keeps expanding. Suddenly, the idea of having all dependencies live in the same monorepo seems less crazy. There are still no great monorepo tools coming up to 2021 for mobile, so you’d have to build much of this for yourself.
At Uber, we moved to a monorepo for Android and one for iOS, when we were around 100 engineers on both platforms. Should you do the same? Unfortunately, there is no definite answer, only tradeoffs to consider. What is more important for your team? Short build times, standardized versioning or keeping the investment low on custom tooling or mobile platform teams?
Keeping master green - at scale. How difficult is it to keep the time to merge changes short, and the main branch always green? If you have builds that execute fast - say, 15 minutes to run all tests - and few merges per day - say, 10 or 20 - this is no challenge.
What if your build takes 30 minutes to complete with unit, integration and UI tests? And what if you have 10 pull requests coming in, per hour? Is there a way you can both keep the main branch green, and also keep the time to merge low - as close as possible to the 30 minutes minimum?
To make this happen, you need to parallelize both the builds. But is it safe to run two PRs parallel, when one could impact the other one? This problem leads to complex, but equally exciting territory. At Uber, we ended up building a Submit Queue, that breaks up builds into parallel parts, then does probabilistic modelling on the build queue, determining changes likely to pass, and prioritizing these. If this sounds something you’d like to learn more about, you can read the details of this approach in this whitepaper.
While there are probably few companies with the amount of mobile PRs happening daily as Uber had: if you’re one of them, you’ll have to invest in bringing down the average time to merge changes, and keep the engineering feedback loops low.
You are reading an early draft from the book Building Mobile Apps at Scale. For the final, expanded and improved content, grab the book now - it's free to download as a PDF until 31 May.
Building Mobile Apps at Scale
"An essential read for anyone working with mobile apps. Not just for mobile engineers - but also on the backend or web teams. The book is full of insights coming from someone who has done engineering at scale."
- Ruj Sabya, formerly Sr Engineering Manager @ Flipkart