你可以使用 ML Kit 偵測及追蹤影片畫面中的物件。
傳遞 ML Kit 圖片時,ML Kit 會為每張圖片傳回清單,最多列出五個偵測到的物件,以及這些物件在圖片中的位置。偵測影片串流中的物件時,每個物件都有 ID,可用來追蹤所有圖片的物件。您也可以選擇啟用粗略的物件分類,將物件加上廣泛類別說明。
事前準備
- 如果您尚未將 Firebase 新增至 Android 專案,請先完成這項操作。
- 將 ML Kit Android 程式庫的依附元件新增至模組 (應用程式層級) Gradle 檔案 (通常是
app/build.gradle
):apply plugin: 'com.android.application' apply plugin: 'com.google.gms.google-services' dependencies { // ... implementation 'com.google.firebase:firebase-ml-vision:24.0.3' implementation 'com.google.firebase:firebase-ml-vision-object-detection-model:19.0.6' }
1. 設定物件偵測工具
如要開始偵測及追蹤物件,請先建立 FirebaseVisionObjectDetector
的執行個體,並選擇性地指定您要從預設值變更的任何偵測工具設定。
請使用
FirebaseVisionObjectDetectorOptions
物件,根據您的用途設定物件偵測工具。您可以變更下列設定:物件偵測器設定 偵測模式 STREAM_MODE
(預設) |SINGLE_IMAGE_MODE
在
STREAM_MODE
(預設) 中,物件偵測工具會在低延遲的情況下執行,但可能會在偵測工具的前幾次叫用時產生不完整的結果 (例如未指定的定界框或類別標籤)。此外,在STREAM_MODE
中,偵測工具會將追蹤 ID 指派給物件,讓您用來跨影格追蹤物件。當您想追蹤物件,或覺得低延遲度很重要時 (例如即時處理影片串流),請使用這個模式。在
SINGLE_IMAGE_MODE
中,物件偵測工具會等到偵測到物件的定界框,以及 (如果您已啟用分類) 類別標籤可供使用,然後才會傳回結果。因此,偵測延遲時間可能會更長。此外,SINGLE_IMAGE_MODE
中不會指派追蹤 ID。如果延遲時間不重要,而且您不想處理部分結果,請使用這個模式。偵測並追蹤多個物件 false
(預設) |true
可指定最多偵測及追蹤五個物件,還是僅偵測最顯眼的物件 (預設)。
將物件分類 false
(預設) |true
是否要將偵測到的物件歸類為粗略的類別。 啟用後,物件偵測工具會將物件分為下列類別:時尚商品、食品、居家用品、地點、植物和不明。
物件偵測和追蹤 API 已針對下列兩種核心用途進行最佳化:
- 在相機觀景窗中即時偵測及追蹤最顯眼的物件
- 偵測靜態圖片中的多個物體
如何針對這些用途設定 API:
Java
// Live detection and tracking FirebaseVisionObjectDetectorOptions options = new FirebaseVisionObjectDetectorOptions.Builder() .setDetectorMode(FirebaseVisionObjectDetectorOptions.STREAM_MODE) .enableClassification() // Optional .build(); // Multiple object detection in static images FirebaseVisionObjectDetectorOptions options = new FirebaseVisionObjectDetectorOptions.Builder() .setDetectorMode(FirebaseVisionObjectDetectorOptions.SINGLE_IMAGE_MODE) .enableMultipleObjects() .enableClassification() // Optional .build();
Kotlin+KTX
// Live detection and tracking val options = FirebaseVisionObjectDetectorOptions.Builder() .setDetectorMode(FirebaseVisionObjectDetectorOptions.STREAM_MODE) .enableClassification() // Optional .build() // Multiple object detection in static images val options = FirebaseVisionObjectDetectorOptions.Builder() .setDetectorMode(FirebaseVisionObjectDetectorOptions.SINGLE_IMAGE_MODE) .enableMultipleObjects() .enableClassification() // Optional .build()
取得
FirebaseVisionObjectDetector
的例項:Java
FirebaseVisionObjectDetector objectDetector = FirebaseVision.getInstance().getOnDeviceObjectDetector(); // Or, to change the default settings: FirebaseVisionObjectDetector objectDetector = FirebaseVision.getInstance().getOnDeviceObjectDetector(options);
Kotlin+KTX
val objectDetector = FirebaseVision.getInstance().getOnDeviceObjectDetector() // Or, to change the default settings: val objectDetector = FirebaseVision.getInstance().getOnDeviceObjectDetector(options)
2. 執行物件偵測工具
如要偵測及追蹤物件,請將圖片傳遞至 FirebaseVisionObjectDetector
執行個體的 processImage()
方法。
針對連續影片或圖片影格,執行下列操作:
從您的圖片建立
FirebaseVisionImage
物件。-
如要從
media.Image
物件建立FirebaseVisionImage
物件 (例如從裝置相機擷取圖片),請將media.Image
物件和圖片的旋轉角度傳遞至FirebaseVisionImage.fromMediaImage()
。如果您使用 CameraX 程式庫,
OnImageCapturedListener
和ImageAnalysis.Analyzer
類別會為您計算旋轉值,因此只要在呼叫FirebaseVisionImage.fromMediaImage()
之前,將旋轉角度轉換為 ML Kit 的ROTATION_
常數即可:Java
private class YourAnalyzer implements ImageAnalysis.Analyzer { private int degreesToFirebaseRotation(int degrees) { switch (degrees) { case 0: return FirebaseVisionImageMetadata.ROTATION_0; case 90: return FirebaseVisionImageMetadata.ROTATION_90; case 180: return FirebaseVisionImageMetadata.ROTATION_180; case 270: return FirebaseVisionImageMetadata.ROTATION_270; default: throw new IllegalArgumentException( "Rotation must be 0, 90, 180, or 270."); } } @Override public void analyze(ImageProxy imageProxy, int degrees) { if (imageProxy == null || imageProxy.getImage() == null) { return; } Image mediaImage = imageProxy.getImage(); int rotation = degreesToFirebaseRotation(degrees); FirebaseVisionImage image = FirebaseVisionImage.fromMediaImage(mediaImage, rotation); // Pass image to an ML Kit Vision API // ... } }
Kotlin+KTX
private class YourImageAnalyzer : ImageAnalysis.Analyzer { private fun degreesToFirebaseRotation(degrees: Int): Int = when(degrees) { 0 -> FirebaseVisionImageMetadata.ROTATION_0 90 -> FirebaseVisionImageMetadata.ROTATION_90 180 -> FirebaseVisionImageMetadata.ROTATION_180 270 -> FirebaseVisionImageMetadata.ROTATION_270 else -> throw Exception("Rotation must be 0, 90, 180, or 270.") } override fun analyze(imageProxy: ImageProxy?, degrees: Int) { val mediaImage = imageProxy?.image val imageRotation = degreesToFirebaseRotation(degrees) if (mediaImage != null) { val image = FirebaseVisionImage.fromMediaImage(mediaImage, imageRotation) // Pass image to an ML Kit Vision API // ... } } }
如果您使用的相機程式庫不提供圖像旋轉功能,您可依據裝置旋轉情形和裝置相機感應器方向計算曝光:
Java
private static final SparseIntArray ORIENTATIONS = new SparseIntArray(); static { ORIENTATIONS.append(Surface.ROTATION_0, 90); ORIENTATIONS.append(Surface.ROTATION_90, 0); ORIENTATIONS.append(Surface.ROTATION_180, 270); ORIENTATIONS.append(Surface.ROTATION_270, 180); } /** * Get the angle by which an image must be rotated given the device's current * orientation. */ @RequiresApi(api = Build.VERSION_CODES.LOLLIPOP) private int getRotationCompensation(String cameraId, Activity activity, Context context) throws CameraAccessException { // Get the device's current rotation relative to its "native" orientation. // Then, from the ORIENTATIONS table, look up the angle the image must be // rotated to compensate for the device's rotation. int deviceRotation = activity.getWindowManager().getDefaultDisplay().getRotation(); int rotationCompensation = ORIENTATIONS.get(deviceRotation); // On most devices, the sensor orientation is 90 degrees, but for some // devices it is 270 degrees. For devices with a sensor orientation of // 270, rotate the image an additional 180 ((270 + 270) % 360) degrees. CameraManager cameraManager = (CameraManager) context.getSystemService(CAMERA_SERVICE); int sensorOrientation = cameraManager .getCameraCharacteristics(cameraId) .get(CameraCharacteristics.SENSOR_ORIENTATION); rotationCompensation = (rotationCompensation + sensorOrientation + 270) % 360; // Return the corresponding FirebaseVisionImageMetadata rotation value. int result; switch (rotationCompensation) { case 0: result = FirebaseVisionImageMetadata.ROTATION_0; break; case 90: result = FirebaseVisionImageMetadata.ROTATION_90; break; case 180: result = FirebaseVisionImageMetadata.ROTATION_180; break; case 270: result = FirebaseVisionImageMetadata.ROTATION_270; break; default: result = FirebaseVisionImageMetadata.ROTATION_0; Log.e(TAG, "Bad rotation value: " + rotationCompensation); } return result; }
Kotlin+KTX
private val ORIENTATIONS = SparseIntArray() init { ORIENTATIONS.append(Surface.ROTATION_0, 90) ORIENTATIONS.append(Surface.ROTATION_90, 0) ORIENTATIONS.append(Surface.ROTATION_180, 270) ORIENTATIONS.append(Surface.ROTATION_270, 180) } /** * Get the angle by which an image must be rotated given the device's current * orientation. */ @RequiresApi(api = Build.VERSION_CODES.LOLLIPOP) @Throws(CameraAccessException::class) private fun getRotationCompensation(cameraId: String, activity: Activity, context: Context): Int { // Get the device's current rotation relative to its "native" orientation. // Then, from the ORIENTATIONS table, look up the angle the image must be // rotated to compensate for the device's rotation. val deviceRotation = activity.windowManager.defaultDisplay.rotation var rotationCompensation = ORIENTATIONS.get(deviceRotation) // On most devices, the sensor orientation is 90 degrees, but for some // devices it is 270 degrees. For devices with a sensor orientation of // 270, rotate the image an additional 180 ((270 + 270) % 360) degrees. val cameraManager = context.getSystemService(CAMERA_SERVICE) as CameraManager val sensorOrientation = cameraManager .getCameraCharacteristics(cameraId) .get(CameraCharacteristics.SENSOR_ORIENTATION)!! rotationCompensation = (rotationCompensation + sensorOrientation + 270) % 360 // Return the corresponding FirebaseVisionImageMetadata rotation value. val result: Int when (rotationCompensation) { 0 -> result = FirebaseVisionImageMetadata.ROTATION_0 90 -> result = FirebaseVisionImageMetadata.ROTATION_90 180 -> result = FirebaseVisionImageMetadata.ROTATION_180 270 -> result = FirebaseVisionImageMetadata.ROTATION_270 else -> { result = FirebaseVisionImageMetadata.ROTATION_0 Log.e(TAG, "Bad rotation value: $rotationCompensation") } } return result }
然後將
media.Image
物件和旋轉值傳遞至FirebaseVisionImage.fromMediaImage()
:Java
FirebaseVisionImage image = FirebaseVisionImage.fromMediaImage(mediaImage, rotation);
Kotlin+KTX
val image = FirebaseVisionImage.fromMediaImage(mediaImage, rotation)
- 如要從檔案 URI 建立
FirebaseVisionImage
物件,請將應用程式結構定義和檔案 URI 傳遞至FirebaseVisionImage.fromFilePath()
。使用ACTION_GET_CONTENT
意圖提示使用者從圖片庫應用程式中選取圖片時,這項功能就很實用。Java
FirebaseVisionImage image; try { image = FirebaseVisionImage.fromFilePath(context, uri); } catch (IOException e) { e.printStackTrace(); }
Kotlin+KTX
val image: FirebaseVisionImage try { image = FirebaseVisionImage.fromFilePath(context, uri) } catch (e: IOException) { e.printStackTrace() }
- 如要從
ByteBuffer
或位元組陣列建立FirebaseVisionImage
物件,請先按照上述的media.Image
輸入方式計算圖片旋轉角度。接著,建立
FirebaseVisionImageMetadata
物件,其中包含圖片的高度、寬度、顏色編碼格式和旋轉:Java
FirebaseVisionImageMetadata metadata = new FirebaseVisionImageMetadata.Builder() .setWidth(480) // 480x360 is typically sufficient for .setHeight(360) // image recognition .setFormat(FirebaseVisionImageMetadata.IMAGE_FORMAT_NV21) .setRotation(rotation) .build();
Kotlin+KTX
val metadata = FirebaseVisionImageMetadata.Builder() .setWidth(480) // 480x360 is typically sufficient for .setHeight(360) // image recognition .setFormat(FirebaseVisionImageMetadata.IMAGE_FORMAT_NV21) .setRotation(rotation) .build()
使用緩衝區或陣列和中繼資料物件建立
FirebaseVisionImage
物件:Java
FirebaseVisionImage image = FirebaseVisionImage.fromByteBuffer(buffer, metadata); // Or: FirebaseVisionImage image = FirebaseVisionImage.fromByteArray(byteArray, metadata);
Kotlin+KTX
val image = FirebaseVisionImage.fromByteBuffer(buffer, metadata) // Or: val image = FirebaseVisionImage.fromByteArray(byteArray, metadata)
- 如要從
Bitmap
物件建立FirebaseVisionImage
物件,請按照下列步驟操作:Java
FirebaseVisionImage image = FirebaseVisionImage.fromBitmap(bitmap);
Kotlin+KTX
val image = FirebaseVisionImage.fromBitmap(bitmap)
Bitmap
物件代表的圖片必須直立,無需額外旋轉。
-
將圖片傳遞至
processImage()
方法:Java
objectDetector.processImage(image) .addOnSuccessListener( new OnSuccessListener<List<FirebaseVisionObject>>() { @Override public void onSuccess(List<FirebaseVisionObject> detectedObjects) { // Task completed successfully // ... } }) .addOnFailureListener( new OnFailureListener() { @Override public void onFailure(@NonNull Exception e) { // Task failed with an exception // ... } });
Kotlin+KTX
objectDetector.processImage(image) .addOnSuccessListener { detectedObjects -> // Task completed successfully // ... } .addOnFailureListener { e -> // Task failed with an exception // ... }
如果呼叫
processImage()
成功,系統會將FirebaseVisionObject
清單傳遞至成功事件監聽器。每個
FirebaseVisionObject
都包含下列屬性:定界框 Rect
:表示物件在圖片中的位置。追蹤 ID 一個整數,可在圖片中識別物件。SINGLE_IMAGE_MODE 中有空值。 類別 物件的概略類別。如果物件偵測工具未啟用分類功能,這個屬性一律為 FirebaseVisionObject.CATEGORY_UNKNOWN
。可信度 物件分類的可信度值。如果物件偵測工具未啟用分類功能,或物件遭歸類為不明,一律為 null
。Java
// The list of detected objects contains one item if multiple object detection wasn't enabled. for (FirebaseVisionObject obj : detectedObjects) { Integer id = obj.getTrackingId(); Rect bounds = obj.getBoundingBox(); // If classification was enabled: int category = obj.getClassificationCategory(); Float confidence = obj.getClassificationConfidence(); }
Kotlin+KTX
// The list of detected objects contains one item if multiple object detection wasn't enabled. for (obj in detectedObjects) { val id = obj.trackingId // A number that identifies the object across images val bounds = obj.boundingBox // The object's position in the image // If classification was enabled: val category = obj.classificationCategory val confidence = obj.classificationConfidence }
提升可用性和效能
為獲得最佳使用者體驗,請在應用程式中遵循以下規範:
- 是否成功偵測物件,取決於物件的視覺複雜度。如果物件只有少量視覺特徵,可能需要佔用較大的圖片部分才能偵測。您必須為使用者提供指引,瞭解如何擷取能與要偵測的物件類型搭配運作的輸入內容。
- 使用分類功能時,如果您想偵測無法正常歸入支援類別的物件,可以針對未知物件實作特殊處理。
此外,您也可以參考 [ML Kit Material Design 展示應用程式][showcase-link]{: .external },以及 Material Design 採用機器學習技術的功能模式系列文章。
在即時應用程式中使用串流模式時,請遵循下列準則,以達到最佳的影格速率:
在串流模式下,請勿使用多項物件偵測功能,因為大多數裝置無法產生適當的影格速率。
如果不需要分類功能,請停用分類功能。
- 限制對偵測工具的呼叫。如果在偵測工具執行時有新的影片影格,請捨棄影格。
- 如果您使用偵測工具的輸出內容,在輸入圖片上疊加圖像,請先從 ML Kit 取得結果,然後透過一個步驟算繪圖像和疊加層。這樣一來,每個輸入影格就只會算繪到顯示介面一次。
-
如果你使用 Camera2 API,請擷取
ImageFormat.YUV_420_888
格式的圖片。如果您使用舊版 Camera API,請拍攝
ImageFormat.NV21
格式的圖片。