Integrate Test Lab into your CI/CD system

1. Introduction

Last Updated: 2022-04-07

How to run mobile tests with CI/CD systems

Running mobile tests can be difficult: many apps run on different platforms, devices, and API versions. App developers want to test on as many configurations as possible to catch issues before their users do. But, cost and resource constraints limit the number of test devices and the amount of manual maintenance that individual developers can invest in. When the development process scales up, especially for continuous integration/continuous development (CI/CD) systems, the testing process needs to be automated while minimizing cost and effort of maintenance.

To help you understand how your apps perform on your users' devices, Firebase Test Lab hosts a range of mobile physical and virtual devices in our data centers. We also provide the Google Cloud CLI, a platform independent, command-line tool that orchestrates running tests on the devices in our data centers. The gcloud CLI makes it simple to integrate Test Lab's cloud-based testing solution into existing CI/CD workflows.

Cross-platform support

Test Lab provides testing solutions for Android and iOS apps and specialty support for mobile games like Unity. Testing options cover popular testing frameworks like Android Espresso, UI Automator, and iOS XCTest. Using our automated crawler, Robo, Test Lab can even run tests without any test code.

No hardware requirements

Test Lab hosts physical devices in Google data centers, and it hosts virtual devices in Google Cloud. All you need to do is send your tests to Test Lab and wait for the results.

Fast and reliable

Running many, simultaneous test cases can take a long time, blocking CI/CD processes. With Test Lab, you can easily shard tests and run them on multiple devices in parallel. You can also detect flakiness, a common mobile testing pain point.

What you'll learn

How to build test artifacts
How to run a mobile test using the gcloud CLI
How to set up Jenkins CI
How to run mobile tests using Jenkins CI
How to configure tests to scale up with CI systems

This codelab focuses on running tests. Non-relevant concepts and code blocks are glossed over and are provided for you to simply copy and paste.

What you'll need

If you prefer to build things from scratch, you need Android Studio to run an Android test, or XCode to run an iOS test. Or just bring yourself and we will provide the artifacts.

2. Run a test with the gcloud CLI

Build an Android app APK and testing APK

To run a test with Test Lab, begin by building an Android app APK and a testing APK, which contains instrumented tests to run on hardware devices or emulators. If you already have a working codebase, you can build your own APKs or you can use the BasicSample for Espresso.

To build APKs using Gradle commands, you must install the Android SDK. If your Android Studio isn't installed on your machine, install Android Studio and Android SDK, and set ANDROID_HOME environment with your Android SDK directory. For example, in your ~/.bash_profile file, add the following line:

export ANDROID_HOME=~/Android/Sdk # For linux
export ANDROID_HOME=~/Library/Android/sdk  # For MacOS

Then, run the following commands to clone the code and build APKs:

git clone https://github.com/android/testing-samples.git
cd testing-samples/ui/espresso/BasicSample/
./gradlew assembleDebug  # for generating app apk
./gradlew assembleDebugAndroidTest # for generating testing apk

After running the clone and build commands, you can find the app APK and testing APK in the following locations:

app/build/outputs/apk/debug/app-debug.apk
app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk

Alternatively, you can use the following options to get the testing and app APKs:

To build APKs in Android Studio, follow the instructions in Test in Android Studio.
Download the sample Notepad app. Use the binary file app-debug-unaligned.apk and the instrumentation tests file app-debug-test-unaligned.apk, which are located in NotePad/app/build/outputs/apk/.

If you have working source code for an iOS app, you can write an XCTest and build a zip file from the app and the tests.

Use the gcloud CLI to run a test with Test Lab

In this section, you create a Firebase project and configure your local Google Cloud SDK environment. To learn more, see Start testing with the gcloud CLI .

Set up the gcloud CLI

Download the Google Cloud SDK, which includes the gcloud CLI tool.
Verify that your installation is up to date:

gcloud components update

gcloud auth login

Set your Firebase project in the gcloud CLI, where PROJECT_ID is your Firebase project ID. You can find the project ID in the Firebase Console URL, which follows this naming convention: https://console.firebase.google.com/project/[PROJECT_ID]/...

gcloud config set project PROJECT_ID

If you don't have a Firebase project, create one in the Firebase Console.

Run a test using the built APKs

In this section, you run an instrumentation test on Test Lab's default device, which is a Pixel 3 with API level 28. If you'd like to use a different device, check available devices.

Using the app and testing APKs that you built in Build an Android app APK and testing APK, run an Android Instrumentation test using the following command:

gcloud firebase test android run \
  --type instrumentation \
  --app app/build/outputs/apk/debug/app-debug.apk \
  --test app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk

Analyze the test results

You can analyze the test results using any of the following options:

gcloud summary - when your test finishes, the gcloud CLI prints a basic summary of your test results.
gcloud exit code - after the test completes, the command exits with 0 if the test passed. If the test fails, the command exits with a non-zero exit code.
Firebase Console - the gcloud CLI prints a link to the Firebase Console. The link follows the naming convention https://console.firebase.google.com/project/PROJECT_ID/testlab/... To learn more, see Interpret results from a single test execution.
JUnit XML file - the gcloud CLI prints the following:

Raw results will be stored in your Cloud Storage bucket at [https://console.developers.google.com/storage/browser/test-lab-xxxxx/xxxxx/]

test-lab-xxxxx/xxxxx is the Cloud Storage bucket that stores test artifacts and results. To locate the JUnit XML file that contains the test results, open the link and navigate to blueline-28-en-portrait/test_result_1.xml.

3. Set up with Jenkins CI

In this section you use Jenkins CI, a popular CI system, to run tests with Test Lab. If you want to use a different CI system, see Advanced best practices for running large test suites and the documentation for other CI systems like Bitrise and Circle CI. You can use Flank as an alternative to gcloud CLI.

Enable the gcloud CLI for Jenkins

Before you can use Test Lab with Jenkins, you must enable required APIs and set up a service account that Jenkins can use to authenticate with the gcloud CLI.

Add a Google Cloud service account for Jenkins

Service accounts are limited accounts intended for service integrations. These accounts offer fine-grained control for specific services and aren't subject to spam checks or captcha prompts, which could otherwise block your CI builds.

To create a service account, follow these steps:

Open the Service Accounts page in the Google Cloud Console.
Click Create service account, add a Name and Description, and then click Create and continue.
From the Select a role dropdown, select Basic, and then select Editor.
Click Continue, then click Done.

Next, you create and download an authentication key that Jenkins can use to authenticate as the service account you created.

To create and download the service account key, follow these steps:

From the Service Accounts page in the Google Cloud Console, click the email associated with the account you created.
Select Keys, then click Add key and Create new key.
Select JSON, then click Create.
When you are prompted to download the file, click OK. Download the file to a safe place on your computer. You need this file later when configuring Jenkins.

To learn more about creating service accounts, see Creating a service account.

Enable required Google Cloud APIs

The Cloud Testing API lets you run tests on Google infrastructure. You enabled this API when you completed Run a test with the gcloud CLI. The Cloud Tool Results API lets you programmatically access your test results.

Open the Google Developers Console API Library.
In the search bar at the top of the Firebase Console, enter the name of each required API ( Cloud Testing API and Cloud Tool Results API). The overview page for the requested API appears.
Click Enable API on each API's overview page.

Install and set up Jenkins

You can install and set up Jenkins CI on Linux, macOS, Windows, and many other environments. Certain details of this codelab are specific to installing and running Jenkins CI on Linux, including the use of slashes (/) in file paths.

To download and install Jenkins on a computer running Linux or Windows, follow the instructions for installing Jenkins. After you install Jenkins, follow the same Jenkins installation instructions to complete setup and to access the Jenkins dashboard using http://localhost:8080.

Verify installed plugins

Jenkins supports different version control systems. In this codelab, you use Git to run the previous test. And to get a better experience with running the gcloud CLI, you need to install the GCloud SDK Plugin.

On the Jenkins dashboard, click Manage Jenkins and then click Manage Plugins.
Search for the Git and GCloud SDK plugins and install them (if they are not installed yet).

Configure the location of your Android and Google Cloud SDKs

You now instruct Jenkins where to find the Google Cloud SDK and Android SDK.

To configure the Google Cloud and Android SDKs for Jenkins, follow these steps:

On the Jenkins dashboard, click Manage Jenkins, and then click Global Tool Configuration.
in the Google Cloud SDK section, click Add Google Cloud SDK.
In the Name field, enter a name for the Google Cloud SDK instance that's easy to remember, for example, GCloud-SDK.
Enter your Google Cloud SDK home directory, for example, /opt/google-cloud-sdk.
Click Save.
Configure your system-wide properties for the Android and Google Cloud SDKs by opening Dashboard > Manage Jenkins > Configure System.
Select the Environment variables checkbox and click Add.
In the Name field, enter ANDROID_HOME. In the Value field, enter the location of your Android SDK, for example, /opt/Android/Sdk.
Click the Tool Locations checkbox and click Add. From the Name dropdown, select the name of the Google Cloud SDK instance you added in Global Tool Configuration.
In the Home field, enter the location of your Google Cloud SDK, for example, /opt/google-cloud-sdk.
Click Save.

Add your service account credentials to Jenkins

You now add your gcloud CLI service account credentials to Jenkins so that Jenkins can authenticate and successfully run gcloud CLI commands.

On the Jenkins dashboard, click Manage Jenkins and then click Manage Credentials.
In the Stores scoped to Jenkins section, click the (global) domain link, then click Add Credentials.
From the Kind dropdown, select Google Service Account from private key.
In the Project Name field, enter your Firebase project name.
Select JSON key, then click Browse and browse to the location where you saved your service account key.
Click Create.

You are now ready to configure your Jenkins automated build for Test Lab.

4. Run Test Lab tests with Jenkins CI

Now that you've set up Jenkins, let's put everything together and run the same test using Jenkins.

You can configure Jenkins to run automated builds and to run tests each time updates to your app are checked in. You can also configure Jenkins to run builds periodically. To learn more about configuring builds in Jenkins, see Configuring automatic builds.

Create and configure your Jenkins project

Create a project to run continuous integration testing of your app with Test Lab.

Create a Jenkins project

Open the Jenkins dashboard by browsing to http://localhost:8080.
On the Jenkins dashboard, click New Item.
Enter a name for your project in the Item name field, and click Freestyle project to create a project that uses a single build configuration.
Click OK. To complete project configurations, complete the remaining sections in sequence..

Configure source code management

From the Source Code Management tab, select Git.
Enter https://github.com/android/testing-samples.git in the Repository URL field.

Configure the build environment

The Build Environment section lets you configure build characteristics. Enable Google Cloud authentication by following these steps:

In the Build Environment tab, select the GCloud SDK authentication checkbox and select the installation name that you chose when you added your Google Cloud installation information.
From the Google Credentials dropdown, select the service account credentials you configured.

Add Gradle build steps to rebuild APK packages

Scroll down to the Build section, click Add build step, and then select Execute shell.
Add a build step to run the following commands in the main directory for your application:

cd ui/espresso/BasicSample/
./gradlew assembleDebug  # for generating app apk
./gradlew assembleDebugAndroidTest # for generating testing apk

Add a gcloud CLI build step to Jenkins

Now you are ready to add a build step to Jenkins to run Test Lab using the gcloud CLI command line.

To add the gcloud CLI build step, add a new Execute shell build step and enter the following command:

gcloud config set project PROJECT_ID
gcloud firebase test android run \
--app ${WORKSPACE}/ui/espresso/BasicSample/app/build/outputs/apk/debug/app-debug.apk
--test ${WORKSPACE}/ui/espresso/BasicSample/app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk

After finishing the project setup, scroll down to the bottom of the page and save the configuration. You are redirected to the home page for the new project.

Run tests with Jenkins

Now you are ready to build the Jenkins project and run a test like the manual test you ran with the gcloud CLI.

To run test with Jenkins, follow these steps:

To trigger the build manually, click Build Now.
Check the progress and results by clicking the new build > Console Output.

If you don't see errors, congratulations! You just set up a project and ran a test on a Pixel 2 device with one click. If you encounter errors, see Troubleshooting.

5. Advanced best practices for running large test suites

When you want to run many, simultaneous tests, follow these best practices to ensure that your tests run fast and reliably.

Increase coverage on multiple device configurations

You can easily add multiple devices to the gcloud CLI build step. And you can check and pick available devices, versions, locales, and orientations. For example, the following command runs tests on two devices:

A virtual Google Pixel 2 with API level 30 in portrait orientation and the English locale
A physical Samsung Galaxy S20 with API level 29 in landscape orientation and the French locale

gcloud firebase test android run \
  --type instrumentation \
  --app app-debug.apk \
  --test app-debug-test.apk \
  --device model=Pixel2,version=30,locale=en,orientation=portrait  \
  --device model=x1q,version=29,locale=fr,orientation=landscape

Use a gcloud CLI YAML configuration file

If you would rather manage your arguments in one place or with your source control system, you can specify these arguments in a YAML-formatted argument file. To learn how to use this feature, run gcloud topic arg-files.

Shard tests to run in parallel

Test sharding divides a set of tests into subgroups (shards) that run separately in isolation. Test Lab automatically runs each shard in parallel using multiple devices, which lets Test Lab complete the entire set of tests more quickly. For more information, see Enable sharding.

To enable sharding, use the gcloud beta channel, and add either –num-uniform-shards or –test-targets-for-shard flag to the gcloud CLI build step. For example, to shard your tests into five executions and run in parallel, run the following commands:

gcloud beta firebase test android run \
  --type instrumentation \
  --app app-debug.apk \
  --test app-debug-test.apk \
  --num-uniform-shards 5

Enable flakiness detection

Mobile tests can often be flaky. Sometimes tests pass, and other times they might fail, even with the same setup. You can detect whether a failed test was flaky by using the –num-flaky-test-attempts gcloud CLI flag. This flag specifies the number of times a test execution should be re-attempted if one or more of the test cases fail for any reason.

For example, to rerun failed test cases three more times after the initial failed run, specify the following:

gcloud beta firebase test android run \
  --type instrumentation \
  --app app-debug.apk \
  --test app-debug-test.apk \
  --num-flaky-test-attempts 3

6. Congratulations

Congratulations, you successfully ran your first Test Lab test with a CI system!

You built app and testing APKs, and you ran an instrumentation test with Test Lab using the gcloud CLI. You also set up a Jenkins project to automate the same test.

Now you know the key steps to integrate Test Lab into your CI/CD system.

What's next?

Check out How to set up CI using the Firebase Emulator Suite