Get started with instrumentation tests

This guide describes how to prepare and run an instrumentation test using Firebase Test Lab. To use this guide, you'll need an instrumentation test (written by you or your team) that uses the Espresso or UI Automator Android test frameworks. Instrumentation tests can run up to 45 minutes on physical devices and up to 60 minutes on virtual devices.

In the steps later, you'll upload your app's APK and your test's APK to Firebase.

Note: Test Lab no longer supports the Robotium Android test framework.

(Optional) Add the screenshot library to your app

Firebase Test Lab includes a library (testlab-instr-lib) that you can use to process any screenshots you take with AndroidX’s ScreenCapture when running instrumentation tests, such as tests written using the Espresso test framework. This section describes how to create ScreenCapture objects with the AndroidX library and how to process them using testlab-instr-lib.

After your instrumentation test has run, you can view the captured screenshots in the Firebase console.

Try out a sample app

Download the NotePad sample app to try out this functionality. The ability to take screenshots is already incorporated into the NotePad project.

Step 1. Add the screenshot library to your project

1. In your test project's root-level settings Gradle file (settings.gradle.kts or settings.gradle), add Google's Maven repository to every repositories section:

   pluginManagement {
       repositories {
           // Add the following line:
           google() // Google's Maven repository
           mavenCentral()
           gradlePluginPortal()
       }
   }
   dependencyResolutionManagement {
       repositoriesMode.set(RepositoriesMode.FAIL_ON_PROJECT_REPOS)
       repositories {
           // Add the following line:
           google() // Google's Maven repository
           mavenCentral()
       }
   }
   // ...

2. In your module (app-level) Gradle file (usually <project>/<app-module>/build.gradle.kts or <project>/<app-module>/build.gradle), add a dependency for the Test Lab screenshot library.

   dependencies {
     // ...
     // Add Test Lab's instrumentation test screenshot library:
     androidTestImplementation("com.google.firebase:testlab-instr-lib:0.2")
     // ...

3. In your test's AndroidManifest.xml file, register the FirebaseScreenCaptureProcessor in a meta-data tag within the <instrumentation> element. You can also specify the processor as an argument in AndroidJUnitRunner instead (see the AndroidJUnitRunner reference documentation for instructions on how).

   <instrumentation
     // Check that you have the following line (if not, add it):
     android:name="androidx.test.runner.AndroidJUnitRunner" // Specifies AndroidJUnitRunner as the test runner
     android:targetPackage="com.your.package.name">
   
   // Add the following:
   <meta-data
     android:name="screenCaptureProcessors"
     android:value="com.google.firebase.testlab.screenshot.FirebaseScreenCaptureProcessor" />
   </instrumentation>
   ...
   

4. In your app's AndroidManifest.xml file, add the following lines within the <manifest> element:

    <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>
   

5. In your AndroidManifest.xml file, specify system permissions for your app by adding the following lines within the <manifest> tag. If you're testing on Android 10 (API level 29) or higher, omit the WRITE_EXTERNAL_STORAGE permission (your app does not require this permission in order to read and write screenshots to the device).

   <manifest ... >
       <!-- WRITE_EXTERNAL_STORAGE is not needed on Android 10 (API level 29) or higher. -->
       <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>
       <uses-permission android:name="android.permission.INTERNET"/>
       ...
   </manifest>

Step 2. Take screenshots during your test

At any point in your test where you want to take a screenshot, call the Screenshot.capture() method from the AndroidX library. This produces a ScreenCapture object. When you call process() on the ScreenCapture object, it gets processed using the ScreenCaptureProcessor that's registered in your AndroidManifest.xml. Note that the BasicScreenCaptureProcessor is used if no processors are registered. Since you registered the FirebaseScreenCaptureProcessor, your screenshots will be processed via FirebaseScreenCaptureProcessor and will be available for you with your results when you run your test with Firebase Test Lab.

Example use cases for creating a ScreenCapture:

• Take a full ScreenCapture on a API Build.VERSION_CODES.JELLY_BEAN_MR2 and above:

  Screenshot.capture()
  

• Take a ScreenCapture of the Activity on any API level. Note this is the only option for devices that are below Build.VERSION_CODES.JELLY_BEAN_MR2.

  @Rule
    public ActivityTestRule<MainActivity> activityRule = new ActivityTestRule<>(MainActivity.class);
  ...
  Screenshot.capture(activityRule.getActivity());
  ...
  

Example use cases for processing a ScreenCapture

• Process a ScreenCapture via the FirebaseScreenCaptureProcessor:

  Screenshot.capture().process();
  

• Process a ScreenCapture via a specified ScreenCaptureProcessor (this allows you to skip registering the processor):

  Set<ScreenCaptureProcessor> processors = new HashSet<>();
  processors.add(new FirebaseScreenCaptureProcessor());
  Screenshot.capture().process(processors);
  

• Set the name and format of the ScreenCapture and process it using the registered processor:

  Screenshot.capture().setName("myscreenshot").setFormat(CompressFormat.JPEG).process();
  

Step 3. Build and run your test

1. Build your app and test APKs (see Test your app for instructions).

2. Upload the APK files to the Test Lab dashboard of the Firebase console.

3. Finally, run your test.

Step 4. View your test screenshots

After your test has completed, you can view any screenshots taken in the Firebase console.

1. In the Tests tab, select your completed test, then click the Results tab.

2. Select your test again, then click the Screenshots tab that appears.

(Optional) Enable additional test features

You can enable the following features in your test before running it with Test Lab:

Enable Orchestrator

Android Test Orchestrator is a tool that runs each of your app's instrumentation tests independently. Test Lab always uses the latest version of Orchestrator.

To enable Orchestrator for Test Lab, in instrumentation test setup, click Additional options > Run with Orchestrator.

Caution: To avoid spending quota on or being billed for malfunctioning tests, run Orchestrator locally before trying it in Test Lab. Confirm that Orchestrator is working in your app by running a test on your own machine before uploading your APK. Keep in mind that testing with Orchestrator takes slightly longer than without it, and it might impact your billing or cause your tests to exceed your timeout limit.

When you use Orchestrator, you benefit from the following:

• No shared state. Each test runs in its own instrumentation instance, so a shared state doesn't accumulate across tests.
• Isolated crashes. If a test crashes, only that instrumentation is terminated, and other tests in your suite can still run.

Keep in mind that when you use Orchestrator, each test runs its own instrumentation instance, which means that the app process is restarted after every test case. The resulting increased run times might impact your quota usage or billed time and might cause you to exceed your devices' timeout limits. If you reduce your app's startup time, this overhead will shorten.

To set additional options for Orchestrator, specify them via environmentVariables field. For example, to use clearPackageData, use this option in gcloud:

--environment-variables clearPackageData=true

Enable sharding

Test sharding divides a set of tests into sub-groups (shards) that run separately in isolation. Test Lab automatically runs each shard in parallel using multiple devices, and completes the entire set of tests in less time.

For example, if you create N shards, for each device you select, Test Lab spins up N identical devices and runs a subset of the tests on each device. This means that sharded test cases can result in multiple test executions per device. Non-sharded test cases, however, result in one test execution per device. To learn Test Lab concepts, see Key concepts.

To enable test sharding in the Firebase console, follow these steps:

1. In instrumentation test setup, click Additional options.

2. In the Sharding section, enter the number of shards you want to run.

Billing for test shards

Test Lab implements your shards by leveraging AndroidJUnitRunner's built-in sharding mechanism. To avoid being charged for spinning up empty shards (shards without assigned test cases), the number of shards you create should be less than the total number of test cases. Depending on how long each test case takes to run, it's typically a good idea to assign 2-10 test cases per shard.

Caution: Sharding tests may cost more. Although test sharding reduces the time it takes for all your tests to run, the overhead of starting your app on each shard may increase individual device times (the time it takes for a device to run its tests). Because charges are calculated on a per-minute basis for each shard, your total charges may increase. In addition, shards are counted towards your daily quota for test executions.

For more information on billing, read Usage, quotas, and billing.