使用 Firebase ML 识别地标 (Android)

您可以使用 Firebase ML 识别图片中的知名地标。

准备工作

  1. 将 Firebase 添加到您的 Android 项目(如果尚未添加)。
  2. 在您的模块(应用级)Gradle 文件(通常是 <project>/<app-module>/build.gradle.kts<project>/<app-module>/build.gradle)中,添加 Firebase ML Vision Android 库的依赖项。我们建议使用 Firebase Android BoM 来实现库版本控制。
    dependencies {
        // Import the BoM for the Firebase platform
        implementation(platform("com.google.firebase:firebase-bom:32.5.0"))
    
        // Add the dependency for the Firebase ML Vision library
        // When using the BoM, you don't specify versions in Firebase library dependencies
        implementation 'com.google.firebase:firebase-ml-vision'
    }
    

    借助 Firebase Android BoM,可确保您的应用使用的始终是 Firebase Android 库的兼容版本。

    (替代方法) 在不使用 BoM 的情况下添加 Firebase 库依赖项

    如果您选择不使用 Firebase BoM,则必须在每个 Firebase 库的依赖项行中指定相应的库版本。

    请注意,如果您在应用中使用多个 Firebase 库,我们强烈建议您使用 BoM 来管理库版本,从而确保所有版本都兼容。

    dependencies {
        // Add the dependency for the Firebase ML Vision library
        // When NOT using the BoM, you must specify versions in Firebase library dependencies
        implementation 'com.google.firebase:firebase-ml-vision:24.1.0'
    }
    
  3. 如果您尚未为项目启用基于 Cloud 的 API,请立即按照以下步骤启用:

    1. 打开 Firebase 控制台的 Firebase ML API 页面
    2. 如果您尚未将项目升级到 Blaze 定价方案,请点击升级以执行此操作。(只有在您的项目未采用 Blaze 方案时,系统才会提示您进行升级。)

      只有 Blaze 级项目才能使用基于 Cloud 的 API。

    3. 如果尚未启用基于 Cloud 的 API,请点击启用基于 Cloud 的 API

配置地标检测器

默认情况下,Cloud 检测器使用 STABLE 版本的模型,最多会返回 10 个结果。如需更改这两项设置中的任何一项,请使用 FirebaseVisionCloudDetectorOptions 对象进行指定。

例如,如需更改这两项默认设置,请按照以下示例构建 FirebaseVisionCloudDetectorOptions 对象:

Kotlin+KTX

val options = FirebaseVisionCloudDetectorOptions.Builder()
    .setModelType(FirebaseVisionCloudDetectorOptions.LATEST_MODEL)
    .setMaxResults(15)
    .build()

Java

FirebaseVisionCloudDetectorOptions options =
        new FirebaseVisionCloudDetectorOptions.Builder()
                .setModelType(FirebaseVisionCloudDetectorOptions.LATEST_MODEL)
                .setMaxResults(15)
                .build();

如需使用默认设置,可以在下一步中使用 FirebaseVisionCloudDetectorOptions.DEFAULT

运行地标检测器

如需识别图片中的地标,请基于设备上的以下资源创建 FirebaseVisionImage 对象:Bitmapmedia.ImageByteBuffer、字节数组或文件。然后,将 FirebaseVisionImage 对象传递给 FirebaseVisionCloudLandmarkDetectordetectInImage 方法。

  1. 基于图片创建 FirebaseVisionImage 对象。

    • 如需基于 media.Image 对象创建 FirebaseVisionImage 对象(例如从设备的相机捕获图片时),请将 media.Image 对象和图片的旋转角度传递给 FirebaseVisionImage.fromMediaImage()

      如果您使用了 CameraX 库,OnImageCapturedListenerImageAnalysis.Analyzer 类会为您计算旋转角度值,因此您只需在调用 FirebaseVisionImage.fromMediaImage() 之前将旋转角度转换为 Firebase ML 的 ROTATION_ 常量:

      Kotlin+KTX

      private class YourImageAnalyzer : ImageAnalysis.Analyzer {
          private fun degreesToFirebaseRotation(degrees: Int): Int = when(degrees) {
              0 -> FirebaseVisionImageMetadata.ROTATION_0
              90 -> FirebaseVisionImageMetadata.ROTATION_90
              180 -> FirebaseVisionImageMetadata.ROTATION_180
              270 -> FirebaseVisionImageMetadata.ROTATION_270
              else -> throw Exception("Rotation must be 0, 90, 180, or 270.")
          }
      
          override fun analyze(imageProxy: ImageProxy?, degrees: Int) {
              val mediaImage = imageProxy?.image
              val imageRotation = degreesToFirebaseRotation(degrees)
              if (mediaImage != null) {
                  val image = FirebaseVisionImage.fromMediaImage(mediaImage, imageRotation)
                  // Pass image to an ML Vision API
                  // ...
              }
          }
      }
      

      Java

      private class YourAnalyzer implements ImageAnalysis.Analyzer {
      
          private int degreesToFirebaseRotation(int degrees) {
              switch (degrees) {
                  case 0:
                      return FirebaseVisionImageMetadata.ROTATION_0;
                  case 90:
                      return FirebaseVisionImageMetadata.ROTATION_90;
                  case 180:
                      return FirebaseVisionImageMetadata.ROTATION_180;
                  case 270:
                      return FirebaseVisionImageMetadata.ROTATION_270;
                  default:
                      throw new IllegalArgumentException(
                              "Rotation must be 0, 90, 180, or 270.");
              }
          }
      
          @Override
          public void analyze(ImageProxy imageProxy, int degrees) {
              if (imageProxy == null || imageProxy.getImage() == null) {
                  return;
              }
              Image mediaImage = imageProxy.getImage();
              int rotation = degreesToFirebaseRotation(degrees);
              FirebaseVisionImage image =
                      FirebaseVisionImage.fromMediaImage(mediaImage, rotation);
              // Pass image to an ML Vision API
              // ...
          }
      }
      

      如果您没有使用可提供图片旋转角度的相机库,可以根据设备的旋转角度和设备中相机传感器的朝向来计算旋转角度:

      Kotlin+KTX

      private val ORIENTATIONS = SparseIntArray()
      
      init {
          ORIENTATIONS.append(Surface.ROTATION_0, 90)
          ORIENTATIONS.append(Surface.ROTATION_90, 0)
          ORIENTATIONS.append(Surface.ROTATION_180, 270)
          ORIENTATIONS.append(Surface.ROTATION_270, 180)
      }
      /**
       * Get the angle by which an image must be rotated given the device's current
       * orientation.
       */
      @RequiresApi(api = Build.VERSION_CODES.LOLLIPOP)
      @Throws(CameraAccessException::class)
      private fun getRotationCompensation(cameraId: String, activity: Activity, context: Context): Int {
          // Get the device's current rotation relative to its "native" orientation.
          // Then, from the ORIENTATIONS table, look up the angle the image must be
          // rotated to compensate for the device's rotation.
          val deviceRotation = activity.windowManager.defaultDisplay.rotation
          var rotationCompensation = ORIENTATIONS.get(deviceRotation)
      
          // On most devices, the sensor orientation is 90 degrees, but for some
          // devices it is 270 degrees. For devices with a sensor orientation of
          // 270, rotate the image an additional 180 ((270 + 270) % 360) degrees.
          val cameraManager = context.getSystemService(CAMERA_SERVICE) as CameraManager
          val sensorOrientation = cameraManager
              .getCameraCharacteristics(cameraId)
              .get(CameraCharacteristics.SENSOR_ORIENTATION)!!
          rotationCompensation = (rotationCompensation + sensorOrientation + 270) % 360
      
          // Return the corresponding FirebaseVisionImageMetadata rotation value.
          val result: Int
          when (rotationCompensation) {
              0 -> result = FirebaseVisionImageMetadata.ROTATION_0
              90 -> result = FirebaseVisionImageMetadata.ROTATION_90
              180 -> result = FirebaseVisionImageMetadata.ROTATION_180
              270 -> result = FirebaseVisionImageMetadata.ROTATION_270
              else -> {
                  result = FirebaseVisionImageMetadata.ROTATION_0
                  Log.e(TAG, "Bad rotation value: $rotationCompensation")
              }
          }
          return result
      }

      Java

      private static final SparseIntArray ORIENTATIONS = new SparseIntArray();
      static {
          ORIENTATIONS.append(Surface.ROTATION_0, 90);
          ORIENTATIONS.append(Surface.ROTATION_90, 0);
          ORIENTATIONS.append(Surface.ROTATION_180, 270);
          ORIENTATIONS.append(Surface.ROTATION_270, 180);
      }
      
      /**
       * Get the angle by which an image must be rotated given the device's current
       * orientation.
       */
      @RequiresApi(api = Build.VERSION_CODES.LOLLIPOP)
      private int getRotationCompensation(String cameraId, Activity activity, Context context)
              throws CameraAccessException {
          // Get the device's current rotation relative to its "native" orientation.
          // Then, from the ORIENTATIONS table, look up the angle the image must be
          // rotated to compensate for the device's rotation.
          int deviceRotation = activity.getWindowManager().getDefaultDisplay().getRotation();
          int rotationCompensation = ORIENTATIONS.get(deviceRotation);
      
          // On most devices, the sensor orientation is 90 degrees, but for some
          // devices it is 270 degrees. For devices with a sensor orientation of
          // 270, rotate the image an additional 180 ((270 + 270) % 360) degrees.
          CameraManager cameraManager = (CameraManager) context.getSystemService(CAMERA_SERVICE);
          int sensorOrientation = cameraManager
                  .getCameraCharacteristics(cameraId)
                  .get(CameraCharacteristics.SENSOR_ORIENTATION);
          rotationCompensation = (rotationCompensation + sensorOrientation + 270) % 360;
      
          // Return the corresponding FirebaseVisionImageMetadata rotation value.
          int result;
          switch (rotationCompensation) {
              case 0:
                  result = FirebaseVisionImageMetadata.ROTATION_0;
                  break;
              case 90:
                  result = FirebaseVisionImageMetadata.ROTATION_90;
                  break;
              case 180:
                  result = FirebaseVisionImageMetadata.ROTATION_180;
                  break;
              case 270:
                  result = FirebaseVisionImageMetadata.ROTATION_270;
                  break;
              default:
                  result = FirebaseVisionImageMetadata.ROTATION_0;
                  Log.e(TAG, "Bad rotation value: " + rotationCompensation);
          }
          return result;
      }

      然后,将 media.Image 对象及旋转角度值传递给 FirebaseVisionImage.fromMediaImage()

      Kotlin+KTX

      val image = FirebaseVisionImage.fromMediaImage(mediaImage, rotation)

      Java

      FirebaseVisionImage image = FirebaseVisionImage.fromMediaImage(mediaImage, rotation);
    • 如需基于文件 URI 创建 FirebaseVisionImage 对象,请将应用上下文和文件 URI 传递给 FirebaseVisionImage.fromFilePath()。如果您使用 ACTION_GET_CONTENT Intent 提示用户从图库应用中选择图片,这一操作会非常有用。

      Kotlin+KTX

      val image: FirebaseVisionImage
      try {
          image = FirebaseVisionImage.fromFilePath(context, uri)
      } catch (e: IOException) {
          e.printStackTrace()
      }

      Java

      FirebaseVisionImage image;
      try {
          image = FirebaseVisionImage.fromFilePath(context, uri);
      } catch (IOException e) {
          e.printStackTrace();
      }
    • 如需基于 ByteBuffer 或字节数组创建 FirebaseVisionImage 对象,请先按上述 media.Image 输入的说明计算图片旋转角度。

      然后,创建一个包含图片的高度、宽度、颜色编码格式和旋转角度的 FirebaseVisionImageMetadata 对象:

      Kotlin+KTX

      val metadata = FirebaseVisionImageMetadata.Builder()
          .setWidth(480) // 480x360 is typically sufficient for
          .setHeight(360) // image recognition
          .setFormat(FirebaseVisionImageMetadata.IMAGE_FORMAT_NV21)
          .setRotation(rotation)
          .build()

      Java

      FirebaseVisionImageMetadata metadata = new FirebaseVisionImageMetadata.Builder()
              .setWidth(480)   // 480x360 is typically sufficient for
              .setHeight(360)  // image recognition
              .setFormat(FirebaseVisionImageMetadata.IMAGE_FORMAT_NV21)
              .setRotation(rotation)
              .build();

      使用缓冲区或数组以及元数据对象来创建 FirebaseVisionImage 对象:

      Kotlin+KTX

      val image = FirebaseVisionImage.fromByteBuffer(buffer, metadata)
      // Or: val image = FirebaseVisionImage.fromByteArray(byteArray, metadata)

      Java

      FirebaseVisionImage image = FirebaseVisionImage.fromByteBuffer(buffer, metadata);
      // Or: FirebaseVisionImage image = FirebaseVisionImage.fromByteArray(byteArray, metadata);
    • 如需基于 Bitmap 对象创建 FirebaseVisionImage 对象,请运行以下代码:

      Kotlin+KTX

      val image = FirebaseVisionImage.fromBitmap(bitmap)

      Java

      FirebaseVisionImage image = FirebaseVisionImage.fromBitmap(bitmap);
      Bitmap 对象表示的图片必须保持竖直,不需要额外的旋转。

  2. 获取 FirebaseVisionCloudLandmarkDetector 的一个实例:

    Kotlin+KTX

    val detector = FirebaseVision.getInstance()
        .visionCloudLandmarkDetector
    // Or, to change the default settings:
    // val detector = FirebaseVision.getInstance()
    //         .getVisionCloudLandmarkDetector(options)

    Java

    FirebaseVisionCloudLandmarkDetector detector = FirebaseVision.getInstance()
            .getVisionCloudLandmarkDetector();
    // Or, to change the default settings:
    // FirebaseVisionCloudLandmarkDetector detector = FirebaseVision.getInstance()
    //         .getVisionCloudLandmarkDetector(options);
  3. 最后,将图片传递给 detectInImage 方法:

    Kotlin+KTX

    val result = detector.detectInImage(image)
        .addOnSuccessListener { firebaseVisionCloudLandmarks ->
            // Task completed successfully
            // ...
        }
        .addOnFailureListener { e ->
            // Task failed with an exception
            // ...
        }

    Java

    Task<List<FirebaseVisionCloudLandmark>> result = detector.detectInImage(image)
            .addOnSuccessListener(new OnSuccessListener<List<FirebaseVisionCloudLandmark>>() {
                @Override
                public void onSuccess(List<FirebaseVisionCloudLandmark> firebaseVisionCloudLandmarks) {
                    // Task completed successfully
                    // ...
                }
            })
            .addOnFailureListener(new OnFailureListener() {
                @Override
                public void onFailure(@NonNull Exception e) {
                    // Task failed with an exception
                    // ...
                }
            });

获取识别出的地标的相关信息

如果地标识别操作成功,系统会向成功监听器传递一组 FirebaseVisionCloudLandmark 对象。每个 FirebaseVisionCloudLandmark 对象代表一个在图片中识别出的地标。对于每个地标,您可以获取它在输入图片中的边界坐标、地标名称、地标的经纬度、地标的知识图谱实体 ID(如果有)以及匹配的置信度分数。例如:

Kotlin+KTX

for (landmark in firebaseVisionCloudLandmarks) {
    val bounds = landmark.boundingBox
    val landmarkName = landmark.landmark
    val entityId = landmark.entityId
    val confidence = landmark.confidence

    // Multiple locations are possible, e.g., the location of the depicted
    // landmark and the location the picture was taken.
    for (loc in landmark.locations) {
        val latitude = loc.latitude
        val longitude = loc.longitude
    }
}

Java

for (FirebaseVisionCloudLandmark landmark: firebaseVisionCloudLandmarks) {

    Rect bounds = landmark.getBoundingBox();
    String landmarkName = landmark.getLandmark();
    String entityId = landmark.getEntityId();
    float confidence = landmark.getConfidence();

    // Multiple locations are possible, e.g., the location of the depicted
    // landmark and the location the picture was taken.
    for (FirebaseVisionLatLng loc: landmark.getLocations()) {
        double latitude = loc.getLatitude();
        double longitude = loc.getLongitude();
    }
}

后续步骤