Go to console

Label images with an AutoML-trained model on Android

After you train your own model using AutoML Vision Edge, you can use it in your app to label images.

See the ML Kit quickstart sample on GitHub for an example of this API in use.

Before you begin

  1. If you haven't already, add Firebase to your Android project.
  2. In your project-level build.gradle file, make sure to include Google's Maven repository in both your buildscript and allprojects sections.
  3. Add the dependencies for the ML Kit Android libraries to your module (app-level) Gradle file (usually app/build.gradle):
    dependencies {
      // ...
    
      implementation 'com.google.firebase:firebase-ml-vision:22.0.0'
      implementation 'com.google.firebase:firebase-ml-vision-automl:18.0.0'
    }
    

1. Load the model

ML Kit runs your AutoML-generated models on the device. However, you can configure ML Kit to load your model either remotely from Firebase, from local storage, or both.

By hosting the model on Firebase, you can update the model without releasing a new app version, and you can use Remote Config and A/B Testing to dynamically serve different models to different sets of users.

If you choose to only provide the model by hosting it with Firebase, and not bundle it with your app, you can reduce the initial download size of your app. Keep in mind, though, that if the model is not bundled with your app, any model-related functionality will not be available until your app downloads the model for the first time.

By bundling your model with your app, you can ensure your app's ML features still work when the Firebase-hosted model isn't available.

Configure a Firebase-hosted model source

To use the remotely-hosted model, register a FirebaseRemoteModel object, specifying the name you assigned the model when you published it, and the conditions under which ML Kit should download the model initially and when updates are available. The download conditions can be different for the initial download and for updates.

Java

FirebaseModelDownloadConditions conditions = new FirebaseModelDownloadConditions.Builder()
        .requireWifi()
        .build();
FirebaseRemoteModel remoteModel = new FirebaseRemoteModel.Builder("my_remote_model")
        .enableModelUpdates(true)
        .setInitialDownloadConditions(conditions)
        .setUpdatesDownloadConditions(conditions)
        .build();
FirebaseModelManager.getInstance().registerRemoteModel(remoteModel);

Kotlin

val conditions = FirebaseModelDownloadConditions.Builder()
        .requireWifi()
        .build()
val remoteModel = FirebaseRemoteModel.Builder("my_remote_model")
        .enableModelUpdates(true)
        .setInitialDownloadConditions(conditions)
        .setUpdatesDownloadConditions(conditions)
        .build()
FirebaseModelManager.getInstance().registerRemoteModel(remoteModel)

Configure a local model source

To bundle the model with your app:

  1. Extract the model and its metadata from the zip archive you downloaded from Firebase console. We recommend you use the files as you downloaded them, without modification (including the file names).
  2. Include your model and its metadata files in your app package:

    1. If you don't have an assets folder in your project, create one by right-clicking the app/ folder, then clicking New > Folder > Assets Folder.
    2. Create a sub-folder under the assets folder to contain the model files.
    3. Copy the files model.tflite, dict.txt, and manifest.json to the sub-folder (all three files must be in the same folder).
  3. Add the following to your app's build.gradle file to ensure Gradle doesn’t compress the model file when building the app:
    android {
        // ...
        aaptOptions {
            noCompress "tflite"
        }
    }
    
    The model file will be included in the app package and available to ML Kit as a raw asset.
  4. Register a FirebaseLocalModel object, specifying the path to the model manifest file and assigning the model a name you will use in the next steps:

    Java

    FirebaseLocalModel localModel = new FirebaseLocalModel.Builder("my_local_model")
            .setAssetFilePath("manifest.json")
            .build();
    FirebaseModelManager.getInstance().registerLocalModel(localModel);
    

    Kotlin

    val localModel = FirebaseLocalModel.Builder("my_local_model")
            .setAssetFilePath("manifest.json")
            .build()
    FirebaseModelManager.getInstance().registerLocalModel(localModel)
    

2. Prepare the input image

Then, for each image you want to label, create a FirebaseVisionImage object using one of the options described in this section and pass it to an instance of FirebaseVisionImageLabeler (described in the next section).

You can create a FirebaseVisionImage object from a media.Image object, a file on the device, a byte array, or a Bitmap object:

  • To create a FirebaseVisionImage object from a media.Image object, such as when capturing an image from a device's camera, pass the media.Image object and the image's rotation to FirebaseVisionImage.fromMediaImage().

    If you use the CameraX library, the OnImageCapturedListener and ImageAnalysis.Analyzer classes calculate the rotation value for you, so you just need to convert the rotation to one of ML Kit's ROTATION_ constants before calling FirebaseVisionImage.fromMediaImage():

    Java

    private class YourAnalyzer implements ImageAnalysis.Analyzer {
    
        private int degreesToFirebaseRotation(int degrees) {
            switch (degrees) {
                case 0:
                    return FirebaseVisionImageMetadata.ROTATION_0;
                case 90:
                    return FirebaseVisionImageMetadata.ROTATION_90;
                case 180:
                    return FirebaseVisionImageMetadata.ROTATION_180;
                case 270:
                    return FirebaseVisionImageMetadata.ROTATION_270;
                default:
                    throw new IllegalArgumentException(
                            "Rotation must be 0, 90, 180, or 270.");
            }
        }
    
        @Override
        public void analyze(ImageProxy imageProxy, int degrees) {
            if (imageProxy == null || imageProxy.getImage() == null) {
                return;
            }
            Image mediaImage = imageProxy.getImage();
            int rotation = degreesToFirebaseRotation(degrees);
            FirebaseVisionImage image =
                    FirebaseVisionImage.fromMediaImage(mediaImage, rotation);
            // Pass image to an ML Kit Vision API
            // ...
        }
    }
    

    Kotlin

    private class YourImageAnalyzer : ImageAnalysis.Analyzer {
        private fun degreesToFirebaseRotation(degrees: Int): Int = when(degrees) {
            0 -> FirebaseVisionImageMetadata.ROTATION_0
            90 -> FirebaseVisionImageMetadata.ROTATION_90
            180 -> FirebaseVisionImageMetadata.ROTATION_180
            270 -> FirebaseVisionImageMetadata.ROTATION_270
            else -> throw Exception("Rotation must be 0, 90, 180, or 270.")
        }
    
        override fun analyze(imageProxy: ImageProxy?, degrees: Int) {
            val mediaImage = imageProxy?.image
            val imageRotation = degreesToFirebaseRotation(degrees)
            if (mediaImage != null) {
                val image = FirebaseVisionImage.fromMediaImage(mediaImage, imageRotation)
                // Pass image to an ML Kit Vision API
                // ...
            }
        }
    }
    

    If you don't use a camera library that gives you the image's rotation, you can calculate it from the device's rotation and the orientation of camera sensor in the device:

    Java

    private static final SparseIntArray ORIENTATIONS = new SparseIntArray();
    static {
        ORIENTATIONS.append(Surface.ROTATION_0, 90);
        ORIENTATIONS.append(Surface.ROTATION_90, 0);
        ORIENTATIONS.append(Surface.ROTATION_180, 270);
        ORIENTATIONS.append(Surface.ROTATION_270, 180);
    }
    
    /**
     * Get the angle by which an image must be rotated given the device's current
     * orientation.
     */
    @RequiresApi(api = Build.VERSION_CODES.LOLLIPOP)
    private int getRotationCompensation(String cameraId, Activity activity, Context context)
            throws CameraAccessException {
        // Get the device's current rotation relative to its "native" orientation.
        // Then, from the ORIENTATIONS table, look up the angle the image must be
        // rotated to compensate for the device's rotation.
        int deviceRotation = activity.getWindowManager().getDefaultDisplay().getRotation();
        int rotationCompensation = ORIENTATIONS.get(deviceRotation);
    
        // On most devices, the sensor orientation is 90 degrees, but for some
        // devices it is 270 degrees. For devices with a sensor orientation of
        // 270, rotate the image an additional 180 ((270 + 270) % 360) degrees.
        CameraManager cameraManager = (CameraManager) context.getSystemService(CAMERA_SERVICE);
        int sensorOrientation = cameraManager
                .getCameraCharacteristics(cameraId)
                .get(CameraCharacteristics.SENSOR_ORIENTATION);
        rotationCompensation = (rotationCompensation + sensorOrientation + 270) % 360;
    
        // Return the corresponding FirebaseVisionImageMetadata rotation value.
        int result;
        switch (rotationCompensation) {
            case 0:
                result = FirebaseVisionImageMetadata.ROTATION_0;
                break;
            case 90:
                result = FirebaseVisionImageMetadata.ROTATION_90;
                break;
            case 180:
                result = FirebaseVisionImageMetadata.ROTATION_180;
                break;
            case 270:
                result = FirebaseVisionImageMetadata.ROTATION_270;
                break;
            default:
                result = FirebaseVisionImageMetadata.ROTATION_0;
                Log.e(TAG, "Bad rotation value: " + rotationCompensation);
        }
        return result;
    }

    Kotlin

    private val ORIENTATIONS = SparseIntArray()
    
    init {
        ORIENTATIONS.append(Surface.ROTATION_0, 90)
        ORIENTATIONS.append(Surface.ROTATION_90, 0)
        ORIENTATIONS.append(Surface.ROTATION_180, 270)
        ORIENTATIONS.append(Surface.ROTATION_270, 180)
    }
    /**
     * Get the angle by which an image must be rotated given the device's current
     * orientation.
     */
    @RequiresApi(api = Build.VERSION_CODES.LOLLIPOP)
    @Throws(CameraAccessException::class)
    private fun getRotationCompensation(cameraId: String, activity: Activity, context: Context): Int {
        // Get the device's current rotation relative to its "native" orientation.
        // Then, from the ORIENTATIONS table, look up the angle the image must be
        // rotated to compensate for the device's rotation.
        val deviceRotation = activity.windowManager.defaultDisplay.rotation
        var rotationCompensation = ORIENTATIONS.get(deviceRotation)
    
        // On most devices, the sensor orientation is 90 degrees, but for some
        // devices it is 270 degrees. For devices with a sensor orientation of
        // 270, rotate the image an additional 180 ((270 + 270) % 360) degrees.
        val cameraManager = context.getSystemService(CAMERA_SERVICE) as CameraManager
        val sensorOrientation = cameraManager
                .getCameraCharacteristics(cameraId)
                .get(CameraCharacteristics.SENSOR_ORIENTATION)!!
        rotationCompensation = (rotationCompensation + sensorOrientation + 270) % 360
    
        // Return the corresponding FirebaseVisionImageMetadata rotation value.
        val result: Int
        when (rotationCompensation) {
            0 -> result = FirebaseVisionImageMetadata.ROTATION_0
            90 -> result = FirebaseVisionImageMetadata.ROTATION_90
            180 -> result = FirebaseVisionImageMetadata.ROTATION_180
            270 -> result = FirebaseVisionImageMetadata.ROTATION_270
            else -> {
                result = FirebaseVisionImageMetadata.ROTATION_0
                Log.e(TAG, "Bad rotation value: $rotationCompensation")
            }
        }
        return result
    }c

    Then, pass the media.Image object and the rotation value to FirebaseVisionImage.fromMediaImage():

    Java

    FirebaseVisionImage image = FirebaseVisionImage.fromMediaImage(mediaImage, rotation);

    Kotlin

    val image = FirebaseVisionImage.fromMediaImage(mediaImage, rotation)
  • To create a FirebaseVisionImage object from a file URI, pass the app context and file URI to FirebaseVisionImage.fromFilePath(). This is useful when you use an ACTION_GET_CONTENT intent to prompt the user to select an image from their gallery app.

    Java

    FirebaseVisionImage image;
    try {
        image = FirebaseVisionImage.fromFilePath(context, uri);
    } catch (IOException e) {
        e.printStackTrace();
    }

    Kotlin

    val image: FirebaseVisionImage
    try {
        image = FirebaseVisionImage.fromFilePath(context, uri)
    } catch (e: IOException) {
        e.printStackTrace()
    }
  • To create a FirebaseVisionImage object from a ByteBuffer or a byte array, first calculate the image rotation as described above for media.Image input.

    Then, create a FirebaseVisionImageMetadata object that contains the image's height, width, color encoding format, and rotation:

    Java

    FirebaseVisionImageMetadata metadata = new FirebaseVisionImageMetadata.Builder()
            .setWidth(480)   // 480x360 is typically sufficient for
            .setHeight(360)  // image recognition
            .setFormat(FirebaseVisionImageMetadata.IMAGE_FORMAT_NV21)
            .setRotation(rotation)
            .build();

    Kotlin

    val metadata = FirebaseVisionImageMetadata.Builder()
            .setWidth(480) // 480x360 is typically sufficient for
            .setHeight(360) // image recognition
            .setFormat(FirebaseVisionImageMetadata.IMAGE_FORMAT_NV21)
            .setRotation(rotation)
            .build()

    Use the buffer or array, and the metadata object, to create a FirebaseVisionImage object:

    Java

    FirebaseVisionImage image = FirebaseVisionImage.fromByteBuffer(buffer, metadata);
    // Or: FirebaseVisionImage image = FirebaseVisionImage.fromByteArray(byteArray, metadata);

    Kotlin

    val image = FirebaseVisionImage.fromByteBuffer(buffer, metadata)
    // Or: val image = FirebaseVisionImage.fromByteArray(byteArray, metadata)
  • To create a FirebaseVisionImage object from a Bitmap object:

    Java

    FirebaseVisionImage image = FirebaseVisionImage.fromBitmap(bitmap);

    Kotlin

    val image = FirebaseVisionImage.fromBitmap(bitmap)
    The image represented by the Bitmap object must be upright, with no additional rotation required.

3. Run the image labeler

To label objects in an image, pass the FirebaseVisionImage object to the FirebaseVisionImageLabeler's process() method.

  1. First, get an instance of FirebaseVisionImageLabeler, configuring it with the names of the local and remote models you want to use, and the confidence score threshold you want to require (see Evaluate your model):

    Java

    FirebaseVisionOnDeviceAutoMLImageLabelerOptions labelerOptions =
            new FirebaseVisionOnDeviceAutoMLImageLabelerOptions.Builder()
                    .setLocalModelName("my_local_model")    // Skip to not use a local model
                    .setRemoteModelName("my_remote_model")  // Skip to not use a remote model
                    .setConfidenceThreshold(0)  // Evaluate your model in the Firebase console
                                                // to determine an appropriate value.
                    .build();
    FirebaseVisionImageLabeler labeler =
            FirebaseVision.getInstance().getOnDeviceAutoMLImageLabeler(labelerOptions);
    

    Kotlin

    val labelerOptions = FirebaseVisionOnDeviceAutoMLImageLabelerOptions.Builder()
            .setLocalModelName("my_local_model")    // Skip to not use a local model
            .setRemoteModelName("my_remote_model")  // Skip to not use a remote model
            .setConfidenceThreshold(0)  // Evaluate your model in the Firebase console
                                        // to determine an appropriate value.
            .build()
    val labeler = FirebaseVision.getInstance().getOnDeviceAutoMLImageLabeler(labelerOptions)
    
  2. Recommended: If you didn't configure a locally-bundled model, make sure the remote model has been downloaded to the device.

    When you run a remotely-hosted model, if the model isn't yet available on the device, the call fails and the model is automatically downloaded to the device in the background. When the download is complete, you can run the model successfully.

    If you want to handle the model downloading task more explicitly, you can start the model downloading task and check its status by calling downloadRemoteModelIfNeeded():

    Java

    FirebaseModelManager.getInstance().downloadRemoteModelIfNeeded(remoteModel)
            .addOnSuccessListener(
                new OnSuccessListener<Void>() {
                  @Override
                  public void onSuccess() {
                    // Model downloaded successfully. Okay to use the model.
                  }
                })
            .addOnFailureListener(
                new OnFailureListener() {
                  @Override
                  public void onFailure(@NonNull Exception e) {
                    // Model couldn’t be downloaded or other internal error.
                    // ...
                  }
                });
    

    Kotlin

    FirebaseModelManager.getInstance().downloadRemoteModelIfNeeded(remoteModel)
            .addOnSuccessListener {
                // Model downloaded successfully. Okay to use the model.
            }
            .addOnFailureListener {
                // Model couldn’t be downloaded or other internal error.
                // ...
            }
    

    The downloadRemoteModelIfNeeded() method starts downloading the model if necessary, and calls the success listener when the download finishes. If the model is already available, the method calls the success listener immediately.

  3. Pass the image to the processImage() method:

    Java

    labeler.processImage(image)
            .addOnSuccessListener(new OnSuccessListener<List<FirebaseVisionImageLabel>>() {
                @Override
                public void onSuccess(List<FirebaseVisionImageLabel> labels) {
                    // Task completed successfully
                    // ...
                }
            })
            .addOnFailureListener(new OnFailureListener() {
                @Override
                public void onFailure(@NonNull Exception e) {
                    // Task failed with an exception
                    // ...
                }
            });
    

    Kotlin

    labeler.processImage(image)
            .addOnSuccessListener { labels ->
                // Task completed successfully
                // ...
            }
            .addOnFailureListener { e ->
                // Task failed with an exception
                // ...
            }
    

    If image labeling succeeds, an array of FirebaseVisionImageLabel objects will be passed to the success listener. From each object, you can get information about a feature recognized in the image.

    For example:

    Java

    for (FirebaseVisionImageLabel label: labels) {
        String text = label.getText();
        float confidence = label.getConfidence();
    }
    

    Kotlin

    for (label in labels) {
        val text = label.text
        val confidence = label.confidence
    }
    

Tips to improve real-time performance

  • Throttle calls to the detector. If a new video frame becomes available while the detector is running, drop the frame. See the VisionProcessorBase class in the quickstart sample app for an example.
  • If you are using the output of the detector to overlay graphics on the input image, first get the result from ML Kit, then render the image and overlay in a single step. By doing so, you render to the display surface only once for each input frame. See the CameraSourcePreview and GraphicOverlay classes in the quickstart sample app for an example.
  • If you use the Camera2 API, capture images in ImageFormat.YUV_420_888 format.

    If you use the older Camera API, capture images in ImageFormat.NV21 format.