使用 Imagen 从图片中移除对象

预览版：使用 Firebase AI Logic SDK 访问 Imagen 模型是一项处于预览版阶段的功能，这意味着它不受任何服务等级协议 (SLA) 或弃用政策的约束，并且可能会以不向后兼容的方式更改。

仅当您使用 Vertex AI Gemini API 时，才支持使用 Imagen 进行编辑。目前也仅支持 Android 和 Flutter 应用。今年晚些时候，我们将支持其他平台。

本页介绍了如何使用 Firebase AI Logic SDK 通过 Imagen 使用图像修复功能从图像中移除对象。

修复是一种基于蒙版的修改。蒙版是一种数字叠加层，用于定义您要修改的特定区域。

运作方式：您提供一张原始图片和一张相应的蒙版图片（自动生成或由您提供），其中定义了要移除的对象或主题的蒙版。您还可以选择提供一段文本提示，描述要移除的内容，或者让模型智能检测要移除的对象。然后，模型会移除对象，并使用新的、符合上下文的内容填充该区域。

例如，您可以遮盖一个球，并将其替换为空白墙壁或草地。

跳转到自动生成的遮罩的代码 跳转到提供遮罩的代码

准备工作

仅在将 Vertex AI Gemini API 用作 API 提供方时可用。

如果您尚未完成入门指南，请先完成该指南。该指南介绍了如何设置 Firebase 项目、将应用连接到 Firebase、添加 SDK、为所选的 API 提供程序初始化后端服务，以及创建 ImagenModel 实例。

本指南假设您使用的是最新的 Firebase AI Logic SDK。如果您仍在使用“Vertex AI in Firebase”SDK，请参阅迁移指南。

支持此功能的模型

Imagen 通过其 capability 模型提供图片编辑功能：

• imagen-3.0-capability-001

请注意，对于 Imagen 模型，global 位置不受支持。

使用自动生成的蒙版移除对象

在试用此示例之前，请完成本指南的准备工作部分，以设置您的项目和应用。

以下示例展示了如何使用修复功能从图片中移除内容，并使用自动蒙版生成功能。 您提供原始图片和文本提示，Imagen 会自动检测并创建蒙版区域来修改原始图片。

注意： 自动为图片中的常见对象生成遮罩的另一种方法是使用 ImagenSemanticMask（而不是 ImagenBackgroundMask）并指定对象的对象类别 ID。例如，您可以指定类别 ID 为 67，模型将检测图片中的所有表格，并为其创建遮罩。

Swift

Swift 不支持使用 Imagen 模型进行图片编辑。今年晚些时候再回来查看！

Kotlin

如需移除具有自动生成的遮罩的对象，请指定 ImagenBackgroundMask。使用 editImage() 并将编辑配置设置为使用 ImagenEditMode.INPAINT_REMOVAL。

// Using this SDK to access Imagen models is a Preview release and requires opt-in
@OptIn(PublicPreviewAPI::class)
suspend fun customizeImage() {
    // Initialize the Vertex AI Gemini API backend service
    // Optionally specify the location to access the model (for example, `us-central1`)
    val ai = Firebase.ai(backend = GenerativeBackend.vertexAI(location = "us-central1"))

    // Create an `ImagenModel` instance with an Imagen "capability" model
    val model = ai.imagenModel("imagen-3.0-capability-001")

    // This example assumes 'originalImage' is a pre-loaded Bitmap.
    // In a real app, this might come from the user's device or a URL.
    val originalImage: Bitmap = TODO("Load your original image Bitmap here")

    // Provide the prompt describing the content to be removed.
    val prompt = "ball"

    // Use the editImage API to remove the unwanted content.
    // Pass the original image, the prompt, and an editing configuration.
    val editedImage = model.editImage(
        sources = listOf(
            ImagenRawImage(originalImage),
            ImagenBackgroundMask(), // Use ImagenBackgroundMask() to auto-generate the mask.
        ),
        prompt = prompt,
        // Define the editing configuration for inpainting and insertion.
        config = ImagenEditingConfig(ImagenEditMode.INPAINT_REMOVAL)
    )

    // Process the resulting 'editedImage' Bitmap, for example, by displaying it in an ImageView.
}

Java

如需移除具有自动生成的遮罩的对象，请指定 ImagenBackgroundMask。使用 editImage() 并将编辑配置设置为使用 ImagenEditMode.INPAINT_REMOVAL。

// Initialize the Vertex AI Gemini API backend service
// Optionally specify the location to access the model (for example, `us-central1`)
// Create an `ImagenModel` instance with an Imagen "capability" model
ImagenModel imagenModel = FirebaseAI.getInstance(GenerativeBackend.vertexAI("us-central1"))
        .imagenModel(
                /* modelName */ "imagen-3.0-capability-001");

ImagenModelFutures model = ImagenModelFutures.from(imagenModel);

// This example assumes 'originalImage' is a pre-loaded Bitmap.
// In a real app, this might come from the user's device or a URL.
Bitmap originalImage = null; // TODO("Load your image Bitmap here");

// Provide the prompt describing the content to be removed.
String prompt = "ball";

// Define the list of sources for the editImage call.
// This includes the original image and the auto-generated mask.
ImagenRawImage rawOriginalImage = new ImagenRawImage(originalImage);
ImagenBackgroundMask rawMaskedImage = new ImagenBackgroundMask();  // Use ImagenBackgroundMask() to auto-generate the mask.

// Define the editing configuration for inpainting and removal.
ImagenEditingConfig config = new ImagenEditingConfig.Builder()
        .setEditMode(ImagenEditMode.INPAINT_REMOVAL)
        .build();

// Use the editImage API to remove the unwanted content.
// Pass the original image, the auto-generated masked image, the prompt, and an editing configuration.
Futures.addCallback(model.editImage(Arrays.asList(rawOriginalImage, rawMaskedImage), prompt, config), new FutureCallback<ImagenGenerationResponse>() {
    @Override
    public void onSuccess(ImagenGenerationResponse result) {
        if (result.getImages().isEmpty()) {
            Log.d("ImageEditor", "No images generated");
        }
        Bitmap editedImage = result.getImages().get(0).asBitmap();
        // Process and use the bitmap to display the image in your UI
    }

    @Override
    public void onFailure(Throwable t) {
        // ...
    }
}, Executors.newSingleThreadExecutor());

Web

Web 应用不支持使用 Imagen 模型进行图片编辑。今年晚些时候再回来查看！

Dart

如需移除具有自动生成的遮罩的对象，请指定 ImagenBackgroundMask。使用 editImage() 并将编辑配置设置为使用 ImagenEditMode.inpaintRemoval。

import 'dart:typed_data';
import 'package:firebase_ai/firebase_ai.dart';
import 'package:firebase_core/firebase_core.dart';
import 'firebase_options.dart';

// Initialize FirebaseApp
await Firebase.initializeApp(
  options: DefaultFirebaseOptions.currentPlatform,
);

// Initialize the Vertex AI Gemini API backend service
// Optionally specify a location to access the model (for example, `us-central1`)
final ai = FirebaseAI.vertexAI(location: 'us-central1');

// Create an `ImagenModel` instance with an Imagen "capability" model
final model = ai.imagenModel(model: 'imagen-3.0-capability-001');

TODO - FLUTTER// This example assumes 'originalImage' is a pre-loaded Uint8List.
// In a real app, this might come from the user's device or a URL.
final Uint8List originalImage = Uint8List(0); // TODO: Load your original image data here.

// Provide the prompt describing the content to be removed.
final prompt = 'ball';

try {
  // Use the editImage API to remove the unwanted content.
  // Pass the original image, the prompt, and an editing configuration.
  final response = await model.editImage(
    sources: [
      ImagenRawImage(originalImage),
      ImagenBackgroundMask(),  // Use ImagenBackgroundMask() to auto-generate the mask.
    ],
    prompt,
    // Define the editing configuration for inpainting and removal.
    config: const ImagenEditingConfig(
      editMode: ImagenEditMode.inpaintRemoval,
    ),
  );

  // Process the result.
  if (response.images.isNotEmpty) {
    final editedImage = response.images.first.bytes;
    // Use the editedImage (a Uint8List) to display the image, save it, etc.
    print('Image successfully generated!');
  } else {
    // Handle the case where no images were generated.
    print('Error: No images were generated.');
  }
} catch (e) {
  // Handle any potential errors during the API call.
  print('An error occurred: $e');
}

Unity

Unity 不支持使用 Imagen 模型进行图片编辑。今年晚些时候再回来查看！

使用提供的蒙版移除对象

在试用此示例之前，请完成本指南的准备工作部分，以设置您的项目和应用。

以下示例展示了如何使用修复功能从图片中移除内容，方法是使用您提供的图片中定义的蒙版。您提供原始图片、文本提示和蒙版图片。

如果您提供遮盖图片，则可以选择是否提供文本提示。 Imagen 可以智能检测要从遮盖区域中移除的对象。不过，如果您要移除的对象不明显，或者您只想移除蒙版区域中的特定对象，请提供文本提示，帮助模型移除正确的对象。

Swift

Swift 不支持使用 Imagen 模型进行图片编辑。今年晚些时候再回来查看！

Kotlin

如需移除对象并提供您自己的蒙版图片，请使用蒙版图片指定 ImagenRawMask。使用 editImage() 并将编辑配置设置为使用 ImagenEditMode.INPAINT_REMOVAL。

// Using this SDK to access Imagen models is a Preview release and requires opt-in
@OptIn(PublicPreviewAPI::class)
suspend fun customizeImage() {
    // Initialize the Vertex AI Gemini API backend service
    // Optionally specify the location to access the model (for example, `us-central1`)
    val ai = Firebase.ai(backend = GenerativeBackend.vertexAI(location = "us-central1"))

    // Create an `ImagenModel` instance with an Imagen "capability" model
    val model = ai.imagenModel("imagen-3.0-capability-001")

    // This example assumes 'originalImage' is a pre-loaded Bitmap.
    // In a real app, this might come from the user's device or a URL.
    val originalImage: Bitmap = TODO("Load your original image Bitmap here")

    // This example assumes 'maskImage' is a pre-loaded Bitmap that contains the masked area.
    // In a real app, this might come from the user's device or a URL.
    val maskImage: Bitmap = TODO("Load your masked image Bitmap here")

    // Provide the prompt describing the content to be removed.
    val prompt = "ball"

    // Use the editImage API to remove the unwanted content.
    // Pass the original image, the masked image, the prompt, and an editing configuration.
    val editedImage = model.editImage(
        referenceImages = listOf(
            ImagenRawImage(originalImage.toImagenInlineImage()),
            ImagenRawMask(maskImage.toImagenInlineImage()), // Use ImagenRawMask() to provide your own masked image.
        ),
        prompt = prompt,
        // Define the editing configuration for inpainting and removal.
        config = ImagenEditingConfig(ImagenEditMode.INPAINT_REMOVAL)
    )

    // Process the resulting 'editedImage' Bitmap, for example, by displaying it in an ImageView.
}

Java

如需移除对象并提供您自己的蒙版图片，请使用蒙版图片指定 ImagenRawMask。使用 editImage() 并将编辑配置设置为使用 ImagenEditMode.INPAINT_REMOVAL。

// Initialize the Vertex AI Gemini API backend service
// Optionally specify the location to access the model (for example, `us-central1`)
// Create an `ImagenModel` instance with an Imagen "capability" model
ImagenModel imagenModel = FirebaseAI.getInstance(GenerativeBackend.vertexAI("us-central1"))
        .imagenModel(
                /* modelName */ "imagen-3.0-capability-001");

ImagenModelFutures model = ImagenModelFutures.from(imagenModel);

// This example assumes 'originalImage' is a pre-loaded Bitmap.
// In a real app, this might come from the user's device or a URL.
Bitmap originalImage = null; // TODO("Load your original image Bitmap here");

// This example assumes 'maskImage' is a pre-loaded Bitmap that contains the masked area.
// In a real app, this might come from the user's device or a URL.
Bitmap maskImage = null; // TODO("Load your masked image Bitmap here");

// Provide the prompt describing the content to be removed.
String prompt = "ball";

// Define the list of source images for the editImage call.
ImagenRawImage rawOriginalImage = new ImagenRawImage(originalImage);
ImagenBackgroundMask rawMaskedImage = new ImagenRawMask(maskImage); // Use ImagenRawMask() to provide your own masked image.

// Define the editing configuration for inpainting and removal.
ImagenEditingConfig config = new ImagenEditingConfig.Builder()
        .setEditMode(ImagenEditMode.INPAINT_REMOVAL)
        .build();

// Use the editImage API to remove the unwanted content.
// Pass the original image, the masked image, the prompt, and an editing configuration.
Futures.addCallback(model.editImage(Arrays.asList(rawOriginalImage, rawMaskedImage), prompt, config), new FutureCallback<ImagenGenerationResponse>() {
    @Override
    public void onSuccess(ImagenGenerationResponse result) {
        if (result.getImages().isEmpty()) {
            Log.d("ImageEditor", "No images generated");
        }
        Bitmap editedImage = result.getImages().get(0).asBitmap();
        // Process and use the bitmap to display the image in your UI
    }

    @Override
    public void onFailure(Throwable t) {
        // ...
    }
}, Executors.newSingleThreadExecutor());

Web

Web 应用不支持使用 Imagen 模型进行图片编辑。今年晚些时候再回来查看！

Dart

如需移除对象并提供您自己的蒙版图片，请使用蒙版图片指定 ImagenRawMask。使用 editImage() 并将编辑配置设置为使用 ImagenEditMode.inpaintRemoval。

import 'dart:typed_data';
import 'package:firebase_ai/firebase_ai.dart';
import 'package:firebase_core/firebase_core.dart';
import 'firebase_options.dart';

// Initialize FirebaseApp
await Firebase.initializeApp(
  options: DefaultFirebaseOptions.currentPlatform,
);

// Initialize the Vertex AI Gemini API backend service
// Optionally specify a location to access the model (for example, `us-central1`)
final ai = FirebaseAI.vertexAI(location: 'us-central1');

// Create an `ImagenModel` instance with an Imagen "capability" model
final model = ai.imagenModel(model: 'imagen-3.0-capability-001');

// This example assumes 'originalImage' is a pre-loaded Uint8List.
// In a real app, this might come from the user's device or a URL.
final Uint8List originalImage = Uint8List(0); // TODO: Load your original image data here.

// This example assumes 'maskImage' is a pre-loaded Uint8List that contains the masked area.
// In a real app, this might come from the user's device or a URL.
final Uint8List maskImage = Uint8List(0); // TODO: Load your masked image data here.

// Provide the prompt describing the content to be removed.
final prompt = 'ball';

try {
  // Use the editImage API to remove the unwanted content.
  // Pass the original image, the prompt, and an editing configuration.
  final response = await model.editImage(
    sources: [
      ImagenRawImage(originalImage),
      ImagenRawMask(maskImage), // Use ImagenRawMask() to provide your own masked image.
    ],
    prompt: prompt,
    // Define the editing configuration for inpainting and removal.
    config: const ImagenEditingConfig(
      editMode: ImagenEditMode.inpaintRemoval,
    ),
  );

  // Process the result.
  if (response.images.isNotEmpty) {
    final editedImage = response.images.first.bytes;
    // Use the editedImage (a Uint8List) to display the image, save it, etc.
    print('Image successfully generated!');
  } else {
    // Handle the case where no images were generated.
    print('Error: No images were generated.');
  }
} catch (e) {
  // Handle any potential errors during the API call.
  print('An error occurred: $e');
}

Unity

Unity 不支持使用 Imagen 模型进行图片编辑。今年晚些时候再回来查看！

最佳做法和限制

建议您在编辑图片时扩大遮罩。这有助于平滑编辑的边界，使其看起来更逼真。一般来说，建议使用 1% 或 2% 的扩张值（0.01 或 0.02）。

就您使用 Firebase AI Logic 的体验提供反馈