您可以使用 ML Kit 為圖片中辨識出的物件,使用裝置上的模型或雲端模型加上標籤。如要瞭解每個方法的優點,請參閱總覽。
事前準備
- 如果您尚未將 Firebase 新增至 Android 專案,請先完成這項操作。
- 將 ML Kit Android 程式庫的依附元件新增至模組 (應用程式層級) Gradle 檔案 (通常是
app/build.gradle
):apply plugin: 'com.android.application' apply plugin: 'com.google.gms.google-services' dependencies { // ... implementation 'com.google.firebase:firebase-ml-vision:24.0.3' implementation 'com.google.firebase:firebase-ml-vision-image-label-model:20.0.1' }
-
選用,但建議使用:如果您使用裝置端 API,請將應用程式設為在從 Play 商店安裝應用程式後,自動將機器學習模型下載至裝置。
如要這麼做,請在應用程式的
AndroidManifest.xml
檔案中新增以下宣告:<application ...> ... <meta-data android:name="com.google.firebase.ml.vision.DEPENDENCIES" android:value="label" /> <!-- To use multiple models: android:value="label,model2,model3" --> </application>
如未啟用安裝期間模型下載功能,系統會在您首次執行裝置端偵測工具時下載模型。下載完成前提出的要求不會產生任何結果。 -
如要使用以雲端為基礎的模型,且您尚未為專案啟用雲端式 API,請立即啟用:
- 開啟 Firebase 控制台的 ML Kit API 頁面。
-
如果您尚未將專案升級至 Blaze 定價方案,按一下「升級」即可進行升級 (只有在專案未採用 Blaze 方案時,系統才會提示您升級)。
只有 Blaze 層級的專案可以使用以雲端為基礎的 API。
- 如果雲端型 API 尚未啟用,請點選「啟用雲端式 API」。
如果只想使用裝置端模型,可以略過這個步驟。
您現在可以使用裝置上的模型或雲端模型為圖片加上標籤。
1. 準備輸入圖片
使用圖片建立FirebaseVisionImage
物件。使用 Bitmap
或 JPEG 格式的 media.Image
時,圖片標籤工具執行速度最快,建議盡可能採用這種格式。
-
如要從
media.Image
物件建立FirebaseVisionImage
物件 (例如從裝置相機擷取圖片),請將media.Image
物件和圖片的旋轉角度傳遞至FirebaseVisionImage.fromMediaImage()
。如果您使用 CameraX 程式庫,
OnImageCapturedListener
和ImageAnalysis.Analyzer
類別會為您計算旋轉值,因此只要在呼叫FirebaseVisionImage.fromMediaImage()
之前,將旋轉角度轉換為 ML Kit 的ROTATION_
常數即可:Java
private class YourAnalyzer implements ImageAnalysis.Analyzer { private int degreesToFirebaseRotation(int degrees) { switch (degrees) { case 0: return FirebaseVisionImageMetadata.ROTATION_0; case 90: return FirebaseVisionImageMetadata.ROTATION_90; case 180: return FirebaseVisionImageMetadata.ROTATION_180; case 270: return FirebaseVisionImageMetadata.ROTATION_270; default: throw new IllegalArgumentException( "Rotation must be 0, 90, 180, or 270."); } } @Override public void analyze(ImageProxy imageProxy, int degrees) { if (imageProxy == null || imageProxy.getImage() == null) { return; } Image mediaImage = imageProxy.getImage(); int rotation = degreesToFirebaseRotation(degrees); FirebaseVisionImage image = FirebaseVisionImage.fromMediaImage(mediaImage, rotation); // Pass image to an ML Kit Vision API // ... } }
Kotlin+KTX
private class YourImageAnalyzer : ImageAnalysis.Analyzer { private fun degreesToFirebaseRotation(degrees: Int): Int = when(degrees) { 0 -> FirebaseVisionImageMetadata.ROTATION_0 90 -> FirebaseVisionImageMetadata.ROTATION_90 180 -> FirebaseVisionImageMetadata.ROTATION_180 270 -> FirebaseVisionImageMetadata.ROTATION_270 else -> throw Exception("Rotation must be 0, 90, 180, or 270.") } override fun analyze(imageProxy: ImageProxy?, degrees: Int) { val mediaImage = imageProxy?.image val imageRotation = degreesToFirebaseRotation(degrees) if (mediaImage != null) { val image = FirebaseVisionImage.fromMediaImage(mediaImage, imageRotation) // Pass image to an ML Kit Vision API // ... } } }
如果您使用的相機程式庫不提供圖像旋轉功能,您可依據裝置旋轉情形和裝置相機感應器方向計算曝光:
Java
private static final SparseIntArray ORIENTATIONS = new SparseIntArray(); static { ORIENTATIONS.append(Surface.ROTATION_0, 90); ORIENTATIONS.append(Surface.ROTATION_90, 0); ORIENTATIONS.append(Surface.ROTATION_180, 270); ORIENTATIONS.append(Surface.ROTATION_270, 180); } /** * Get the angle by which an image must be rotated given the device's current * orientation. */ @RequiresApi(api = Build.VERSION_CODES.LOLLIPOP) private int getRotationCompensation(String cameraId, Activity activity, Context context) throws CameraAccessException { // Get the device's current rotation relative to its "native" orientation. // Then, from the ORIENTATIONS table, look up the angle the image must be // rotated to compensate for the device's rotation. int deviceRotation = activity.getWindowManager().getDefaultDisplay().getRotation(); int rotationCompensation = ORIENTATIONS.get(deviceRotation); // On most devices, the sensor orientation is 90 degrees, but for some // devices it is 270 degrees. For devices with a sensor orientation of // 270, rotate the image an additional 180 ((270 + 270) % 360) degrees. CameraManager cameraManager = (CameraManager) context.getSystemService(CAMERA_SERVICE); int sensorOrientation = cameraManager .getCameraCharacteristics(cameraId) .get(CameraCharacteristics.SENSOR_ORIENTATION); rotationCompensation = (rotationCompensation + sensorOrientation + 270) % 360; // Return the corresponding FirebaseVisionImageMetadata rotation value. int result; switch (rotationCompensation) { case 0: result = FirebaseVisionImageMetadata.ROTATION_0; break; case 90: result = FirebaseVisionImageMetadata.ROTATION_90; break; case 180: result = FirebaseVisionImageMetadata.ROTATION_180; break; case 270: result = FirebaseVisionImageMetadata.ROTATION_270; break; default: result = FirebaseVisionImageMetadata.ROTATION_0; Log.e(TAG, "Bad rotation value: " + rotationCompensation); } return result; }
Kotlin+KTX
private val ORIENTATIONS = SparseIntArray() init { ORIENTATIONS.append(Surface.ROTATION_0, 90) ORIENTATIONS.append(Surface.ROTATION_90, 0) ORIENTATIONS.append(Surface.ROTATION_180, 270) ORIENTATIONS.append(Surface.ROTATION_270, 180) } /** * Get the angle by which an image must be rotated given the device's current * orientation. */ @RequiresApi(api = Build.VERSION_CODES.LOLLIPOP) @Throws(CameraAccessException::class) private fun getRotationCompensation(cameraId: String, activity: Activity, context: Context): Int { // Get the device's current rotation relative to its "native" orientation. // Then, from the ORIENTATIONS table, look up the angle the image must be // rotated to compensate for the device's rotation. val deviceRotation = activity.windowManager.defaultDisplay.rotation var rotationCompensation = ORIENTATIONS.get(deviceRotation) // On most devices, the sensor orientation is 90 degrees, but for some // devices it is 270 degrees. For devices with a sensor orientation of // 270, rotate the image an additional 180 ((270 + 270) % 360) degrees. val cameraManager = context.getSystemService(CAMERA_SERVICE) as CameraManager val sensorOrientation = cameraManager .getCameraCharacteristics(cameraId) .get(CameraCharacteristics.SENSOR_ORIENTATION)!! rotationCompensation = (rotationCompensation + sensorOrientation + 270) % 360 // Return the corresponding FirebaseVisionImageMetadata rotation value. val result: Int when (rotationCompensation) { 0 -> result = FirebaseVisionImageMetadata.ROTATION_0 90 -> result = FirebaseVisionImageMetadata.ROTATION_90 180 -> result = FirebaseVisionImageMetadata.ROTATION_180 270 -> result = FirebaseVisionImageMetadata.ROTATION_270 else -> { result = FirebaseVisionImageMetadata.ROTATION_0 Log.e(TAG, "Bad rotation value: $rotationCompensation") } } return result }
然後將
media.Image
物件和旋轉值傳遞至FirebaseVisionImage.fromMediaImage()
:Java
FirebaseVisionImage image = FirebaseVisionImage.fromMediaImage(mediaImage, rotation);
Kotlin+KTX
val image = FirebaseVisionImage.fromMediaImage(mediaImage, rotation)
- 如要從檔案 URI 建立
FirebaseVisionImage
物件,請將應用程式結構定義和檔案 URI 傳遞至FirebaseVisionImage.fromFilePath()
。使用ACTION_GET_CONTENT
意圖提示使用者從圖片庫應用程式中選取圖片時,這項功能就很實用。Java
FirebaseVisionImage image; try { image = FirebaseVisionImage.fromFilePath(context, uri); } catch (IOException e) { e.printStackTrace(); }
Kotlin+KTX
val image: FirebaseVisionImage try { image = FirebaseVisionImage.fromFilePath(context, uri) } catch (e: IOException) { e.printStackTrace() }
- 如要從
ByteBuffer
或位元組陣列建立FirebaseVisionImage
物件,請先按照上述的media.Image
輸入方式計算圖片旋轉角度。接著,建立
FirebaseVisionImageMetadata
物件,其中包含圖片的高度、寬度、顏色編碼格式和旋轉:Java
FirebaseVisionImageMetadata metadata = new FirebaseVisionImageMetadata.Builder() .setWidth(480) // 480x360 is typically sufficient for .setHeight(360) // image recognition .setFormat(FirebaseVisionImageMetadata.IMAGE_FORMAT_NV21) .setRotation(rotation) .build();
Kotlin+KTX
val metadata = FirebaseVisionImageMetadata.Builder() .setWidth(480) // 480x360 is typically sufficient for .setHeight(360) // image recognition .setFormat(FirebaseVisionImageMetadata.IMAGE_FORMAT_NV21) .setRotation(rotation) .build()
使用緩衝區或陣列和中繼資料物件建立
FirebaseVisionImage
物件:Java
FirebaseVisionImage image = FirebaseVisionImage.fromByteBuffer(buffer, metadata); // Or: FirebaseVisionImage image = FirebaseVisionImage.fromByteArray(byteArray, metadata);
Kotlin+KTX
val image = FirebaseVisionImage.fromByteBuffer(buffer, metadata) // Or: val image = FirebaseVisionImage.fromByteArray(byteArray, metadata)
- 如要從
Bitmap
物件建立FirebaseVisionImage
物件,請按照下列步驟操作:Java
FirebaseVisionImage image = FirebaseVisionImage.fromBitmap(bitmap);
Kotlin+KTX
val image = FirebaseVisionImage.fromBitmap(bitmap)
Bitmap
物件代表的圖片必須直立,無需額外旋轉。
2. 設定並執行映像檔標籤工具
如要為圖片中的物件加上標籤,請將FirebaseVisionImage
物件傳遞至 FirebaseVisionImageLabeler
的 processImage
方法。
首先,請取得
FirebaseVisionImageLabeler
的例項。如要使用裝置端圖片標籤工具,請按照下列步驟操作:
Java
FirebaseVisionImageLabeler labeler = FirebaseVision.getInstance() .getOnDeviceImageLabeler(); // Or, to set the minimum confidence required: // FirebaseVisionOnDeviceImageLabelerOptions options = // new FirebaseVisionOnDeviceImageLabelerOptions.Builder() // .setConfidenceThreshold(0.7f) // .build(); // FirebaseVisionImageLabeler labeler = FirebaseVision.getInstance() // .getOnDeviceImageLabeler(options);
Kotlin+KTX
val labeler = FirebaseVision.getInstance().getOnDeviceImageLabeler() // Or, to set the minimum confidence required: // val options = FirebaseVisionOnDeviceImageLabelerOptions.Builder() // .setConfidenceThreshold(0.7f) // .build() // val labeler = FirebaseVision.getInstance().getOnDeviceImageLabeler(options)
如何使用雲端映像檔標籤工具:
Java
FirebaseVisionImageLabeler labeler = FirebaseVision.getInstance() .getCloudImageLabeler(); // Or, to set the minimum confidence required: // FirebaseVisionCloudImageLabelerOptions options = // new FirebaseVisionCloudImageLabelerOptions.Builder() // .setConfidenceThreshold(0.7f) // .build(); // FirebaseVisionImageLabeler labeler = FirebaseVision.getInstance() // .getCloudImageLabeler(options);
Kotlin+KTX
val labeler = FirebaseVision.getInstance().getCloudImageLabeler() // Or, to set the minimum confidence required: // val options = FirebaseVisionCloudImageLabelerOptions.Builder() // .setConfidenceThreshold(0.7f) // .build() // val labeler = FirebaseVision.getInstance().getCloudImageLabeler(options)
接著,將圖片傳遞至
processImage()
方法:Java
labeler.processImage(image) .addOnSuccessListener(new OnSuccessListener<List<FirebaseVisionImageLabel>>() { @Override public void onSuccess(List<FirebaseVisionImageLabel> labels) { // Task completed successfully // ... } }) .addOnFailureListener(new OnFailureListener() { @Override public void onFailure(@NonNull Exception e) { // Task failed with an exception // ... } });
Kotlin+KTX
labeler.processImage(image) .addOnSuccessListener { labels -> // Task completed successfully // ... } .addOnFailureListener { e -> // Task failed with an exception // ... }
3. 取得加上標籤的物件相關資訊
如果圖片標籤作業成功,FirebaseVisionImageLabel
物件清單會傳遞給成功事件監聽器。每個 FirebaseVisionImageLabel
物件都代表已在圖片中加上標籤的內容。您可以取得每個標籤的文字說明、知識圖譜實體 ID (如有),以及比對的可信度分數。例如:
Java
for (FirebaseVisionImageLabel label: labels) {
String text = label.getText();
String entityId = label.getEntityId();
float confidence = label.getConfidence();
}
Kotlin+KTX
for (label in labels) {
val text = label.text
val entityId = label.entityId
val confidence = label.confidence
}
即時效能改善訣竅
如要在即時應用程式中為圖片加上標籤,請遵循下列準則,以便達到最佳的影格速率:
- 限制對圖片標籤人員的呼叫。如果圖片標籤工具執行期間有新的影片影格可供使用,請捨棄該影格。
- 如果您使用圖片標籤人員的輸出內容來疊加輸入圖像上的圖像,請先從 ML Kit 取得結果,然後透過一個步驟算繪圖像和疊加層。這樣一來,每個輸入影格就只會算繪到顯示介面一次。
-
如果你使用 Camera2 API,請擷取
ImageFormat.YUV_420_888
格式的圖片。如果您使用舊版 Camera API,請拍攝
ImageFormat.NV21
格式的圖片。
後續步驟
- 在部署至使用 Cloud API 的正式版應用程式之前,建議先採取一些額外步驟,預防及降低未經授權 API 存取所造成的影響。