Go to console

Label images with an AutoML-trained model on iOS

After you train your own model using AutoML Vision Edge, you can use it in your app to label images.

See the ML Kit quickstart sample on GitHub for an example of this API in use.

Before you begin

  1. If you have not already added Firebase to your app, do so by following the steps in the getting started guide.
  2. Include the ML Kit libraries in your Podfile:
    pod 'Firebase/Analytics'
    pod 'Firebase/MLVision'
    pod 'Firebase/MLVisionAutoML'
    
    After you install or update your project's Pods, be sure to open your Xcode project using its .xcworkspace.
  3. In your app, import Firebase:

    Swift

    import Firebase
    import FirebaseMLCommon

    Objective-C

    @import Firebase;
    @import FirebaseMLCommon;

1. Load the model

ML Kit runs your AutoML-generated models on the device. However, you can configure ML Kit to load your model either remotely from Firebase, from local storage, or both.

By hosting the model on Firebase, you can update the model without releasing a new app version, and you can use Remote Config and A/B Testing to dynamically serve different models to different sets of users.

If you choose to only provide the model by hosting it with Firebase, and not bundle it with your app, you can reduce the initial download size of your app. Keep in mind, though, that if the model is not bundled with your app, any model-related functionality will not be available until your app downloads the model for the first time.

By bundling your model with your app, you can ensure your app's ML features still work when the Firebase-hosted model isn't available.

Configure a Firebase-hosted model source

To use the remotely-hosted model, register a RemoteModel object, specifying the name you assigned the model when you published it, and the conditions under which ML Kit should download the model initially and when updates are available. The download conditions can be different for the initial download and for updates.

Swift

let initialConditions = ModelDownloadConditions(allowsCellularAccess: true,
                                                allowsBackgroundDownloading: true)
let updateConditions = ModelDownloadConditions(allowsCellularAccess: false,
                                               allowsBackgroundDownloading: true)
let remoteModel = RemoteModel(
    name: "your_remote_model",  // The name you assigned in the console.
    allowsModelUpdates: true,
    initialConditions: initialConditions,
    updateConditions: updateConditions
)
ModelManager.modelManager().register(remoteModel)

Objective-C

FIRModelDownloadConditions *initialConditions =
    [[FIRModelDownloadConditions alloc] initWithAllowsCellularAccess:YES
                                         allowsBackgroundDownloading:YES];
FIRModelDownloadConditions *updateConditions =
    [[FIRModelDownloadConditions alloc] initWithAllowsCellularAccess:NO
                                         allowsBackgroundDownloading:YES];
FIRRemoteModel *remoteModel = [[FIRRemoteModel alloc]
          initWithName:@"your_remote_model"  // The name you assigned in the console.
    allowsModelUpdates:YES
     initialConditions:initialConditions
      updateConditions:updateConditions];
[[FIRModelManager modelManager] registerRemoteModel:remoteModel];

Configure a local model source

To bundle the model with your app:

  1. Extract the model and its metadata from the zip archive you downloaded from Firebase console into a folder:
    my_model
      |____dict.txt
      |____manifest.json
      |____model.tflite
    
    All three files must be in the same folder. We recommend you use the files as you downloaded them, without modification (including the file names).
  2. Copy the folder to your Xcode project, taking care to select Create folder references when you do so. The model file and metadata will be included in the app bundle and available to ML Kit.
  3. Register a LocalModel object, specifying the path to the model manifest file and assigning the model a name you will use in the next steps:

    Swift

    guard let manifestPath = Bundle.main.path(forResource: "manifest",
                                              ofType: "json",
                                              inDirectory: "my_model") else { return }
    let localModel = LocalModel(
        name: "your_local_model",  // Assign any name. You'll use it later to load the model.
        path: manifestPath
    )
    ModelManager.modelManager().register(localModel)
    

    Objective-C

    NSString *manifestPath = [NSBundle.mainBundle pathForResource:@"manifest"
                                                           ofType:@"json"
                                                      inDirectory:@"my_model"];
    FIRLocalModel *localModel = [[FIRLocalModel alloc]
        initWithName:@"your_local_model"  // Assign any name. You'll use it later to load the model.
                path:manifestPath];
    [[FIRModelManager modelManager] registerLocalModel:localModel];
    

2. Prepare the input image

Then, for each image you want to label, create a VisionImage object using one of the options described in this section and pass it to an instance of VisionImageLabeler (described in the next section).

Create a VisionImage object using a UIImage or a CMSampleBufferRef.

To use a UIImage:

  1. If necessary, rotate the image so that its imageOrientation property is .up.
  2. Create a VisionImage object using the correctly-rotated UIImage. Do not specify any rotation metadata—the default value, .topLeft, must be used.

    Swift

    let image = VisionImage(image: uiImage)

    Objective-C

    FIRVisionImage *image = [[FIRVisionImage alloc] initWithImage:uiImage];

To use a CMSampleBufferRef:

  1. Create a VisionImageMetadata object that specifies the orientation of the image data contained in the CMSampleBufferRef buffer.

    To get the image orientation:

    Swift

    func imageOrientation(
        deviceOrientation: UIDeviceOrientation,
        cameraPosition: AVCaptureDevice.Position
        ) -> VisionDetectorImageOrientation {
        switch deviceOrientation {
        case .portrait:
            return cameraPosition == .front ? .leftTop : .rightTop
        case .landscapeLeft:
            return cameraPosition == .front ? .bottomLeft : .topLeft
        case .portraitUpsideDown:
            return cameraPosition == .front ? .rightBottom : .leftBottom
        case .landscapeRight:
            return cameraPosition == .front ? .topRight : .bottomRight
        case .faceDown, .faceUp, .unknown:
            return .leftTop
        }
    }

    Objective-C

    - (FIRVisionDetectorImageOrientation)
        imageOrientationFromDeviceOrientation:(UIDeviceOrientation)deviceOrientation
                               cameraPosition:(AVCaptureDevicePosition)cameraPosition {
      switch (deviceOrientation) {
        case UIDeviceOrientationPortrait:
          if (cameraPosition == AVCaptureDevicePositionFront) {
            return FIRVisionDetectorImageOrientationLeftTop;
          } else {
            return FIRVisionDetectorImageOrientationRightTop;
          }
        case UIDeviceOrientationLandscapeLeft:
          if (cameraPosition == AVCaptureDevicePositionFront) {
            return FIRVisionDetectorImageOrientationBottomLeft;
          } else {
            return FIRVisionDetectorImageOrientationTopLeft;
          }
        case UIDeviceOrientationPortraitUpsideDown:
          if (cameraPosition == AVCaptureDevicePositionFront) {
            return FIRVisionDetectorImageOrientationRightBottom;
          } else {
            return FIRVisionDetectorImageOrientationLeftBottom;
          }
        case UIDeviceOrientationLandscapeRight:
          if (cameraPosition == AVCaptureDevicePositionFront) {
            return FIRVisionDetectorImageOrientationTopRight;
          } else {
            return FIRVisionDetectorImageOrientationBottomRight;
          }
        default:
          return FIRVisionDetectorImageOrientationTopLeft;
      }
    }

    Then, create the metadata object:

    Swift

    let cameraPosition = AVCaptureDevice.Position.back  // Set to the capture device you used.
    let metadata = VisionImageMetadata()
    metadata.orientation = imageOrientation(
        deviceOrientation: UIDevice.current.orientation,
        cameraPosition: cameraPosition
    )

    Objective-C

    FIRVisionImageMetadata *metadata = [[FIRVisionImageMetadata alloc] init];
    AVCaptureDevicePosition cameraPosition =
        AVCaptureDevicePositionBack;  // Set to the capture device you used.
    metadata.orientation =
        [self imageOrientationFromDeviceOrientation:UIDevice.currentDevice.orientation
                                     cameraPosition:cameraPosition];
  2. Create a VisionImage object using the CMSampleBufferRef object and the rotation metadata:

    Swift

    let image = VisionImage(buffer: sampleBuffer)
    image.metadata = metadata

    Objective-C

    FIRVisionImage *image = [[FIRVisionImage alloc] initWithBuffer:sampleBuffer];
    image.metadata = metadata;

3. Run the image labeler

To label objects in an image, pass the VisionImage object to the VisionImageLabeler's process() method.

  1. First, get an instance of VisionImageLabeler, configuring it with the names of the local and remote models you want to use, and the confidence score threshold you want to require (see Evaluate your model):

    Swift

    let labelerOptions = VisionOnDeviceAutoMLImageLabelerOptions(
        remoteModelName: "your_remote_model",  // Or nil to not use a remote model
        localModelName: "your_local_model"     // Or nil to not use a bundled model
    )
    labelerOptions.confidenceThreshold = 0  // Evaluate your model in the Firebase console
                                            // to determine an appropriate value.
    let labeler = Vision.vision().onDeviceAutoMLImageLabeler(options: labelerOptions)
    

    Objective-C

    FIRVisionOnDeviceAutoMLImageLabelerOptions *labelerOptions =
        [[FIRVisionOnDeviceAutoMLImageLabelerOptions alloc]
            initWithRemoteModelName:@"my_remote_model"   // Or nil to not use a remote model
                     localModelName:@"my_local_model"];  // Or nil to not use a bundled model
    labelerOptions.confidenceThreshold = 0;  // Evaluate your model in the Firebase console
                                             // to determine an appropriate value.
    FIRVisionImageLabeler *labeler =
        [[FIRVision vision] onDeviceAutoMLImageLabelerWithOptions:labelerOptions];
    
  2. Recommended: If you didn't configure a locally-bundled model, make sure the remote model has been downloaded to the device.

    When you run a remotely-hosted model, if the model isn't yet available on the device, the call fails and the model is automatically downloaded to the device in the background. When the download is complete, you can run the model successfully.

    If you want to handle the model downloading task more explicitly, you can start the model downloading task and check its status by calling download(remoteModel:):

    Swift

    let downloadProgress = ModelManager.modelManager().download(remoteModel)
    
    // ...
    
    if downloadProgress.isFinished {
        // The model is available on the device
    }

    Objective-C

    NSProgress *downloadProgress = [[FIRModelManager modelManager] downloadRemoteModel:remoteModel];
    
    // ...
    
    if (downloadProgress.isFinished) {
      // The model is available on the device
    }

    You can also get the model download status by attaching observers to the default Notification Center. Be sure to use a weak reference to self in the observer block, since downloads can take some time, and the originating object can be freed by the time the download finishes. For example:

    Swift

    NotificationCenter.default.addObserver(
        forName: .firebaseMLModelDownloadDidSucceed,
        object: nil,
        queue: nil
    ) { [weak self] notification in
        guard let strongSelf = self,
            let userInfo = notification.userInfo,
            let model = userInfo[ModelDownloadUserInfoKey.remoteModel.rawValue]
                as? RemoteModel,
            model.name == "your_remote_model"
            else { return }
        // The model was downloaded and is available on the device
    }
    
    NotificationCenter.default.addObserver(
        forName: .firebaseMLModelDownloadDidFail,
        object: nil,
        queue: nil
    ) { [weak self] notification in
        guard let strongSelf = self,
            let userInfo = notification.userInfo,
            let model = userInfo[ModelDownloadUserInfoKey.remoteModel.rawValue]
                as? RemoteModel
            else { return }
        let error = userInfo[ModelDownloadUserInfoKey.error.rawValue]
        // ...
    }

    Objective-C

    __weak typeof(self) weakSelf = self;
    
    [NSNotificationCenter.defaultCenter
        addObserverForName:FIRModelDownloadDidSucceedNotification
                    object:nil
                     queue:nil
                usingBlock:^(NSNotification *_Nonnull note) {
                  if (weakSelf == nil | note.userInfo == nil) {
                    return;
                  }
                  __strong typeof(self) strongSelf = weakSelf;
    
                  FIRRemoteModel *model = note.userInfo[FIRModelDownloadUserInfoKeyRemoteModel];
                  if ([model.name isEqualToString:@"your_remote_model"]) {
                    // The model was downloaded and is available on the device
                  }
                }];
    
    [NSNotificationCenter.defaultCenter
        addObserverForName:FIRModelDownloadDidFailNotification
                    object:nil
                     queue:nil
                usingBlock:^(NSNotification *_Nonnull note) {
                  if (weakSelf == nil | note.userInfo == nil) {
                    return;
                  }
                  __strong typeof(self) strongSelf = weakSelf;
    
                  NSError *error = note.userInfo[FIRModelDownloadUserInfoKeyError];
                }];
  3. Pass the image to the process() method:

    Swift

    labeler.process(image) { labels, error in
        guard error == nil, let labels = labels else { return }
    
        // Task succeeded.
        // ...
    }
    

    Objective-C

    [labeler
        processImage:image
          completion:^(NSArray<FIRVisionImageLabel *> *_Nullable labels, NSError *_Nullable error) {
            if (error != nil || labels == nil) {
              return;
            }
    
            // Task succeeded.
            // ...
          }];
    

    If image labeling succeeds, an array of VisionImageLabel objects will be passed to the completion handler. From each object, you can get information about a feature recognized in the image.

    For example:

    Swift

    for label in labels {
        let labelText = label.text
        let confidence = label.confidence
    }
    

    Objective-C

    for (FIRVisionImageLabel *label in labels) {
      NSString *labelText = label.text;
      NSNumber *confidence = label.confidence;
    }
    

Tips to improve real-time performance

  • Throttle calls to the detector. If a new video frame becomes available while the detector is running, drop the frame.

Next steps

See the ML Kit quickstart sample on GitHub for an example of this API in use.