Find the broken release
before the app store review does.
Datadog Mobile App Testing runs your recorded flows on real iOS and Android devices, on a schedule and from the release pipeline. We implement it on the flows that matter, wire it into CI so a bad build never reaches the store, and run it every month so the tests still pass for the right reasons.
Quick facts
Related
- Synthetic Monitoring, the same idea for web and APIs
- Real User Monitoring, what real users did in the shipped app
- Continuous Testing, tests as a gate in the pipeline
A device lab that never goes home
A web release that breaks can be rolled back in a minute. A mobile release that breaks is in the store, on the phones, with a review queue between you and the fix. Mobile App Testing is the tap-through on a real device that happens before the submit button, every time.
The flow on a real phone. Sign-in, search, checkout, the permissions prompt, recorded once and run on devices Datadog hosts. The bug that only happens on one manufacturer's keyboard shows up on that manufacturer's keyboard.
Every build, not just the release. The pipeline uploads the candidate build and runs the tests against it with datadog-ci. A failure fails the pipeline, so the broken build is a log line rather than a store listing.
Device families, not a device list to maintain. A test can target any recent device of a given make, and it runs on whichever one is free. New models join the family without anyone editing a test.
Test runs linked to real sessions. With the RUM SDK in the app, Datadog can link each mobile test run to the RUM data for that app. A regression in the test can be put next to the regression in the field.
A test on every device, a build uploaded by hand, and a locator that died with the redesign
How it bills. Mobile App Testing is billed per test run on its own meter, separate from the API and browser test meters. A run is up to 25 steps of one test on one device, so a longer test counts as more than one run, and a test against three device families is at least three runs each time it fires. Runs triggered from CI count the same way. Datadog publishes the rates. We publish nothing here, because the price is not the problem. The device list times the schedule is.
The configuration traps. The device picker makes it easy to tick everything, and a test that proves sign-in works on twelve devices every hour proves it eleven times more than anyone needs. Builds uploaded by hand get uploaded when someone remembers, so the tests run against last sprint's build and pass. A recorded test is a list of locators, and a screen redesign that renames them turns a passing test into a failing one with no change in the app. Self-healing locators catch some of this. They do not catch a screen that was removed. None of these raises an error of its own. The first shows up on the invoice, the second as a false sense of safety, and the third as a muted test.
Field noteThe most common gap we see is the build upload living outside the pipeline. A version is uploaded for the first demo and marked as latest. Every test since has run against it, and passed, while every release since went to the store untested.
Field noteThe device list was chosen from what the team carries. The user base is on older, cheaper handsets with less memory and a different default browser, and the one crash that matters never appears on the devices in the test.
Field noteThe pattern we keep finding is a test suite recorded against staging, pointed at a staging backend that is reset nightly. The test account is wiped at 2am, the test fails at 3am, and the on-call rota learns to ignore mobile test alerts within a fortnight.
Field noteThe tests cover iOS because the first recording was done on an iPhone. Android is a large share of the installed base, on far more device models, and it has no tests, because adding a platform felt like a separate project.
Pipeline first. Flows second. Devices from the installed base, not the office.
If mobile testing is not on yet, LaunchPad runs the rollout as a managed project, and the Digital Experience Accelerator covers it alongside RUM and Synthetics where a mobile app is in scope. If it is already on, a Datadog HealthScan reports run counts, failing tests and which build the tests last ran against before we change anything. Either way, the sequence is the same.
- Wire the build upload into the release pipeline. Every candidate build is uploaded and versioned by the pipeline, with the version name taken from the build number. Nobody uploads by hand after week one.
- Agree the flows and the accounts. Which flows a release must pass to ship, and a test account per flow on a backend that is not reset under it.
- Record each flow once, on each platform. iOS and Android recorded separately, with stable identifiers on the elements the test relies on. We ask the app team to add those identifiers where they are missing.
- Choose device families from the installed base. The RUM device breakdown, or the store analytics, decides which makes and OS versions get a run. The office phones do not.
- Gate the pipeline. The release job runs the suite against the uploaded build with datadog-ci and fails on a failure. A scheduled run against the latest version catches backend changes between releases.
- Hand over, in writing. Which flows are tested on which devices, how builds are uploaded, and which accounts the tests use. Who is told when the gate fails, and the run counts on the day we finished.
Access is admin on the Datadog organisation, a pull request into the release pipeline, and a working session with the mobile team for element identifiers. We do not need production shell access to implement Mobile App Testing, and we do not ask for it.
The app ships every fortnight. The tests have to ship with it.
A screen is redesigned. A new OS version changes the permission dialog. A new market means a new device family. Managed Observability is the monthly service that keeps the tests matching the app they test.
Every month, we review the meter. Runs against the device list and the schedule, with CI runs separated from scheduled runs. A device family that no longer appears in the installed base loses its run.
Coverage follows the roadmap. New flows get a recording before they ship. A new platform or framework gets its own suite. A retired screen loses its test instead of failing forever.
The pipeline is kept honest. We check that the version the tests last ran against is the version that last went to the store. When the upload step is removed in a pipeline refactor, we find out from the data, not from a review.
The incident is ours to take. When a scheduled run shows the release in the store has stopped working, Critical Support takes it from there, 24x7, usually to a backend change that nobody told the mobile team about.
We operate the stack. You own the product.
An implementation partner records the flows and hands them over. A managed partner is still there when the onboarding screen is redesigned in March and every test fails on it. We are the second kind. The tests, the pipeline steps and the decisions stay in your Datadog organisation and your repositories, documented, and you can take them in-house whenever you choose.
Datadog is our proof, not our headline. Datadog certified one managed provider globally. We are it. The Powered by Datadog accreditation means Datadog's own engineers reviewed how we instrument, alert, respond and report. We are also a Datadog Advanced Partner, ISO 27001 certified and Cyber Essentials Plus certified, with offices in Cardiff, London and Dublin.
Datadog Mobile App Testing, answered plainly
How is Datadog Mobile App Testing billed?
Mobile App Testing is billed per test run on its own meter, separate from the API and browser test meters in Synthetic Monitoring. A run is up to 25 steps of one test on one real device, so a longer test counts as more than one run, and a test scheduled against three device families is at least three runs each time it fires. Runs triggered from the CI pipeline count on the same meter. We do not publish prices. The number of devices, the schedule and the CI trigger are what we tune.
Does it run on real phones or emulators?
Real devices, hosted by Datadog in the EU or the US. You pick a specific device or a device family, such as any recent Samsung, and the test runs on whatever matching device is free. That matters for the bugs that only happen on one manufacturer's keyboard or one OS version's permission dialog, which an emulator would never show you.
Do we have to upload every build by hand?
No. The first version can be uploaded through the Datadog interface to get started. For anything after that we wire the release pipeline to upload the build and run the tests against it with the datadog-ci command line tool. Every candidate build is then tested before a human looks at it. The tests can also be set to always run against the latest uploaded version.
What is the difference between Mobile App Testing and mobile RUM?
Mobile App Testing runs a scripted flow on a device Datadog hosts, before or after release, whether or not any user is doing the same thing. Mobile RUM records what real users did in the app they installed. One tells you the release candidate can still sign in. The other tells you how many real users could not. If the RUM SDK is in the app, Datadog can link the test runs to the RUM data, so the two sit side by side.
What does Critical Cloud do with Mobile App Testing every month?
Managed Observability runs it every month. We review run counts against the device list and the schedule. We re-record tests that broke because a screen changed rather than because the app did, and add tests for new flows and new platforms. We check the pipeline is still uploading builds, and report what changed. When a test shows a release in the store is broken, Critical Support takes the incident, 24x7.
- Datadog products: every product, and how we implement and run each one
- Synthetic Monitoring: the web and API tests that share the schedule
- Error Tracking: the crashes grouped into issues once the release is out
- Digital Experience Accelerator: RUM, Synthetics and the dashboards in four weeks
Every build tested on a real phone. No build tested by a reviewer first.
Tell us which flows a release must pass and which devices your users carry. We will tell you what the tests cover today, what they should, and what it costs to get there.