Train an image labeling model with AutoML Vision Edge

To train an image labeling model, you provide AutoML Vision Edge with a set of images and corresponding labels. AutoML Vision Edge uses this dataset to train a new model in the cloud, which you can use for on-device image labeling in your app. (See the Overview for general information about this feature.)

Firebase ML's AutoML Vision Edge features are deprecated. Consider using Vertex AI to automatically train ML models, which you can either export as TensorFlow Lite models for on-device use or deploy for cloud-based inference.

AutoML Vision Edge is a Google Cloud service. Use of the service is subject to the Google Cloud Platform License Agreement and Service Specific Terms, and billed accordingly. For billing information, see the AutoML Pricing page.

Important: You can no longer train models with AutoML Vision Edge while on the Spark plan. If you previously trained models while on the Spark plan, your training data and trained models are still accessible from the Firebase console in read-only mode.

Before you begin

• If you don't already have a Firebase or Google Cloud project, create one in the Firebase console.

• Familiarize yourself with the guidelines presented in Inclusive ML guide - AutoML.

• If you just want to try AutoML Vision Edge, and don't have your own training data, download a sample dataset such as one of the following:

  • TensorFlow's official flower image example dataset
  • This American Sign Language (ASL) alphabet dataset

  You can find more datasets hosted on Kaggle.

1. Assemble your training data

First, you need to put together a training dataset of labeled images. Keep the following guidelines in mind:

• The images must be in one of the following formats: JPEG, PNG, GIF, BMP, ICO.

• Each image must be 30MB or smaller. Note that AutoML Vision Edge downscales most images during preprocessing, so there's generally no accuracy benefit to providing very high resolution images.

• Include at least 10, and preferably 100 or more, examples of each label.

• Include multiple angles, resolutions, and backgrounds for each label.

• The training data should be as close as possible to the data on which predictions are to be made. For example, if your use case involves blurry and low-resolution images (such as from a security camera), your training data should be composed of blurry, low-resolution images.

• The models generated by AutoML Vision Edge are optimized for photographs of objects in the real world. They might not work well for X-rays, hand drawings, scanned documents, receipts, and so on.

  Also, the models can't generally predict labels that humans can't assign. So, if a human can't assign labels by looking at the image for 1-2 seconds, the model likely can't be trained to do it either.

If you need help collecting training data, take a look at the open-source Custom Image Classifier app on GitHub.

When you have your training images ready, prepare them to import into Firebase. You have three options:

Option 1: Structured zip archive

Organize your training images into directories, each named after a label and containing images that are examples of that label. Then, compress the directory structure into a zip archive.

The directory names in this zip archive can be up to 32 ASCII characters long and can contain only alphanumeric characters and the underscore character (_).

For example:

my_training_data.zip
  |____accordion
  | |____001.jpg
  | |____002.jpg
  | |____003.jpg
  |____bass_guitar
  | |____hofner.gif
  | |____p-bass.png
  |____clavier
    |____well-tempered.jpg
    |____well-tempered (1).jpg
    |____well-tempered (2).jpg

Option 2: Cloud Storage with CSV index

Upload your training images to Google Cloud Storage and prepare a CSV file listing the URL of each image, and, optionally, the correct labels for each image. This option is helpful when using very large datasets.

For example, upload your images to Cloud Storage, and prepare a CSV file like the following:

gs://your-training-data-bucket/001.jpg,accordion
gs://your-training-data-bucket/002.jpg,accordion
gs://your-training-data-bucket/003.jpg,accordion
gs://your-training-data-bucket/hofner.gif,bass_guitar
gs://your-training-data-bucket/p-bass.png,bass_guitar
gs://your-training-data-bucket/well-tempered.jpg,clavier
gs://your-training-data-bucket/well-tempered%20(1).jpg,clavier
gs://your-training-data-bucket/well-tempered%20(2).jpg,clavier

The images must be stored in a bucket that's part of your Firebase project's corresponding Google Cloud project.

See Preparing your training data in the Cloud AutoML Vision documentation for more information about preparing the CSV file.

Option 3: Unlabeled images

Label your training images in the Firebase console after you upload them, either individually or in an unstructured zip file. See the next step.

2. Train your model

Next, train a model using your images:

1. Open the Vision Datasets page in the Google Cloud console. Select your project when prompted.

2. Click New dataset, provide a name for the dataset, select the type of model you want to train, and click Create dataset.

3. On your dataset's Import tab, upload either a zip archive of your training images or a CSV file containing the Cloud Storage locations you uploaded them to. See Assemble your training data.

4. After the import task completes, use the Images tab to verify the training data and label any unlabeled images.

5. On the Train tab, click Start training.

   1. Name the model and select the Edge model type.

   2. Configure the following training settings, which govern the performance of the generated model:

      Optimize model for...

      The model configuration to use. You can train faster, smaller, models when low latency or small package size are important, or slower, larger, models when accuracy is most important.

      Node hour budget

      The maximum time, in compute hours, to spend training the model. More training time generally results in a more accurate model. Note that training can be completed in less than the specified time if the system determines that the model is optimized and additional training would not improve accuracy. You are billed only for the hours actually used. Typical training times Very small sets 1 hour 500 images 2 hours 1,000 images 3 hours 5,000 images 6 hours 10,000 images 7 hours 50,000 images 11 hours 100,000 images 13 hours 1,000,000 images 18 hours

3. Evaluate your model

When training completes, you can click the Evaluate tab to see performance metrics for the model.

One important use of this page is to determine the confidence threshold that works best for your model. The confidence threshold is the minimum confidence the model must have for it to assign a label to an image. By moving the Confidence threshold slider, you can see how different thresholds affect the model’s performance. Model performance is measured using two metrics: precision and recall.

In the context of image classification, precision is the ratio of the number of images that were correctly labeled to the number of images the model labeled given the selected threshold. When a model has high precision, it assigns labels incorrectly less often (fewer false positives).

Recall is the ratio of the number of images that were correctly labeled to the number of images that had content the model should have been able to label. When a model has high recall, it fails to assign any label less often (fewer false negatives).

Whether you optimize for precision or recall will depend on your use case. See the AutoML Vision beginners' guide and the Inclusive ML guide - AutoML for more information.

When you find a confidence threshold that produces metrics you're comfortable with, make note of it; you will use the confidence threshold to configure the model in your app. (You can use this tool any time to get an appropriate threshold value.)

4. Publish or download your model

If you are satisfied with the model's performance and want to use it in an app, you have three options, from which you can choose any combination: deploy the model for online prediction, publish the model to Firebase, or download the model and bundle it with your app.

Deploy the model

On your dataset's Test & use tab, you can deploy your model for online prediction, which runs your model in the cloud. This option is covered in the Cloud AutoML docs. The docs on this site deal with the remaining two options.

Publish the model

By publishing the model to Firebase, you can update the model without releasing a new app version, and you can use Remote Config and A/B Testing to dynamically serve different models to different sets of users.

This is a beta release of Firebase ML. This API might be changed in backward-incompatible ways and is not subject to any SLA or deprecation policy.

If you choose to only provide the model by hosting it with Firebase, and not bundle it with your app, you can reduce the initial download size of your app. Keep in mind, though, that if the model is not bundled with your app, any model-related functionality will not be available until your app downloads the model for the first time.

To publish your model, you can use either of two methods:

• Download the TF Lite model from your dataset's Test & use page in the Google Cloud console, and then upload the model on the Custom model page of the Firebase console. This is usually the easiest way to publish a single model.
• Publish the model directly from your Google Cloud project to Firebase using the Admin SDK. You can use this method to batch publish several models or to help create automated publishing pipelines.

To publish the model with the Admin SDK model management API:

1. Install and initialize the SDK.

2. Publish the model.

   You will need to specify the model's resource identifier, which is a string that looks like the following example:

   projects/PROJECT_NUMBER/locations/us-central1/models/MODEL_ID

   PROJECT_NUMBER	The project number of the Cloud Storage bucket that contains the model. This might be your Firebase project or another Google Cloud project. You can find this value on the Settings page of the Firebase console or the Google Cloud console dashboard.
   MODEL_ID	The model's ID, which you got from the AutoML Cloud API.

   Python

   # First, import and initialize the SDK.
   
   # Get a reference to the AutoML model
   source = ml.TFLiteAutoMlSource('projects/{}/locations/us-central1/models/{}'.format(
       # See above for information on these values.
       project_number,
       model_id
   ))
   
   # Create the model object
   tflite_format = ml.TFLiteFormat(model_source=source)
   model = ml.Model(
       display_name="example_model",  # This is the name you will use from your app to load the model.
       tags=["examples"],             # Optional tags for easier management.
       model_format=tflite_format)
   
   # Add the model to your Firebase project and publish it
   new_model = ml.create_model(model)
   new_model.wait_for_unlocked()
   ml.publish_model(new_model.model_id)
   

   Node.js

   // First, import and initialize the SDK.
   
   (async () => {
     // Get a reference to the AutoML model. See above for information on these
     // values.
     const automlModel = `projects/${projectNumber}/locations/us-central1/models/${modelId}`;
   
     // Create the model object and add the model to your Firebase project.
     const model = await ml.createModel({
       displayName: 'example_model',  // This is the name you use from your app to load the model.
       tags: ['examples'],  // Optional tags for easier management.
       tfliteModel: { automlModel: automlModel },
     });
   
     // Wait for the model to be ready.
     await model.waitForUnlocked();
   
     // Publish the model.
     await ml.publishModel(model.modelId);
   
     process.exit();
   })().catch(console.error);
   

Download & bundle the model with your app

By bundling your model with your app, you can ensure your app's ML features still work when the Firebase-hosted model isn't available.

If you both publish the model and bundle it with your app, the app will use the latest version available.

To download your model, click TF Lite on your dataset's Test & use page.

Next steps

Now that you have published or downloaded the model, learn how to use the model in your iOS+ and Android apps.