Mistakes are hard to revert
The second most common pain point is caused by mobile apps being distributed as binaries. Once a user updates to a version with a client-side bug, they are stuck with that bug, until a new version is released and users update to this version. There are a few challenges here:
- Apple does not allow updating native code on the fly. Interestingly enough, they do allow this with non-native code like JavaScript: and this is why solutions where business logic is written using JavaScript, bug fixes are pushed to the client are gaining popularity. Solutions include React Native or Cordova with services like Codepush are gaining strong traction. At Uber, we’ve built a homegrown solution among the same lines, as several other companies have done so.
- It takes from hours to days to release a new app version on the store. This is more pronounced for iOS, where a manual app review needs to take place. Historically, every review had the possibility of rejection. As of June 2020, Apple has changed guidelines so bug fixes are no longer be delayed over guideline violations, except for legal issues.
- Users take days to update to the latest version after a new version is published to the app store. This is true even for users with automated updates turned on.
- You can not assume that all users will get this updated version, ever. Some users might have automated updates disabled. Even when they update, they might skip several versions.
As Chuck Rossi who owned much of release engineering at Facebook puts it like this on a Software Engineering Daily podcast episode:
“It was the most terrifying thing to take 10,000 diffs, package it into effectively a bullet, fire that bullet at the horizon and that bullet, once it leaves the barrel, it’s gone. I cannot get it back and it flies flat and true with no friction and no gravity till the heat death of the universe. It’s gone. I can’t fix it."
Fixing a bug on mobile
This means that all previous versions of your app need to be supported, indefinitely - at least in theory, you should do this. The only exception is if you put home-grown controls in place and build a force update mechanism to limit the past versions to support. Android supports in-app updates through the Play Core library. iOS doesn’t have similar native support. We’ll cover force updates in the “Force upgrading” section in Part 3.
Assuming that you have an app with millions of users, what are the steps you can take to minimize bugs shipped to users, or regressions happening in old versions?
- Do thorough testing, at all levels. Automated testing, manual testing, and consider beta testing with easy feedback loops. A common approach at many companies is releasing the beta app to company employees and beta users and having it “bake” for a week, collecting feedback on any issues.
- Have a feature flagging system in place, so you can revert bugs, on the fly. Still, feature flags add further pain points - we’ll discuss these points in the “Feature flag hell” section, later in the series.
- Consider gradual rollouts, with monitoring (TODO: link to monitoring section) to ensure things work as expected.
- Force upgrading is a robust solution - but you’ll need to put one in place, and some customers might churn as a result. We’ll cover this more in Part 5 of the book.
You are reading an early draft from the book Building Mobile Apps at Scale. For the final, expanded and improved content, grab the book now - it's free to download as a PDF until 31 May.
Building Mobile Apps at Scale
"An essential read for anyone working with mobile apps. Not just for mobile engineers - but also on the backend or web teams. The book is full of insights coming from someone who has done engineering at scale."
- Ruj Sabya, formerly Sr Engineering Manager @ Flipkart