在 iOS 上使用 ML Kit 辨識圖片中的文字

你可以使用 ML Kit 辨識圖片中的文字。ML Kit 會為 適合辨識圖片文字的一般用途 API,例如 路標文字,以及經過最佳化調整,可辨識 文件。一般用途 API 同時具備裝置和雲端模型。 文件文字辨識功能僅適用於雲端式模型。詳情請參閱 總覽,方便您比較 包括雲端和裝置端模型

事前準備

  1. 如果尚未將 Firebase 加入應用程式,請按照下列步驟操作: 入門指南中的步驟。
  2. 在 Podfile 中加入 ML Kit 程式庫:
    pod 'Firebase/MLVision', '6.25.0'
    # If using an on-device API:
    pod 'Firebase/MLVisionTextModel', '6.25.0'
    
    敬上 安裝或更新專案的 Pod 後,請務必開啟 Xcode 專案 .xcworkspace
  3. 在應用程式中匯入 Firebase:

    Swift

    import Firebase

    Objective-C

    @import Firebase;
  4. 如要使用雲端模型,且尚未啟用 為專案設定雲端式 API,請立即採用以下做法:

    1. 開啟 ML Kit Firebase 控制台的 API 頁面
    2. 如果您尚未將專案升級至 Blaze 定價方案,請按一下 如要這麼做,請升級。(只有在您的 專案並未採用 Blaze 方案)。

      只有 Blaze 層級的專案可以使用以雲端為基礎的 API。

    3. 如果尚未啟用雲端式 API,請按一下「Enable Cloud-based API」(啟用雲端式 API) API
    ,瞭解如何調查及移除這項存取權。

    如果只想使用裝置端模型,可以略過這個步驟。

現在可以開始辨識圖片中的文字。

輸入圖片規範

  • 為了讓 ML Kit 準確辨識文字,輸入圖片必須包含 以充足的像素資料表示的文字最適合拉丁字母 每個字元至少要有 16x16 像素中文 日文和韓文文字 (只有雲端式 API 支援)。 字元應為 24x24 像素所有語言通常沒有 對字元大於 24x24 像素的特性來說,準確性的優勢在於。

    舉例來說,640x480 的圖片適合掃描名片 圖片會佔滿圖片的整個寬度如何掃描列印的文件 則建議使用 720x1280 像素的圖片。

  • 圖片焦點不佳可能會降低文字辨識的準確度。如果您不 請嘗試重新擷取圖片。

  • 如果您在即時應用程式中辨識文字,您可能也會 想要考慮輸入圖片的整體尺寸較小 圖片處理速度更快,因此為了縮短延遲時間,擷取 較低的解析度 (請留意上述準確率規定) 確保文字盡可能填滿圖片。另請參閱 即時效能改善秘訣


辨識圖片中的文字

為了透過裝置或雲端模型辨識圖片中的文字, 按照下方說明執行文字辨識工具。

1. 執行文字辨識工具

將圖片做為「UIImage」或「CMSampleBufferRef」傳遞至 `VisionTextRecognizer` 的 `process(_:complete:)` 方法:
  1. 如要取得 VisionTextRecognizer 的例項,請呼叫 onDeviceTextRecognizercloudTextRecognizer

    Swift

    如何使用裝置端模型:

    let vision = Vision.vision()
    let textRecognizer = vision.onDeviceTextRecognizer()

    如要使用雲端模型:

    let vision = Vision.vision()
    let textRecognizer = vision.cloudTextRecognizer()
    
    // Or, to provide language hints to assist with language detection:
    // See https://cloud.google.com/vision/docs/languages for supported languages
    let options = VisionCloudTextRecognizerOptions()
    options.languageHints = ["en", "hi"]
    let textRecognizer = vision.cloudTextRecognizer(options: options)

    Objective-C

    如何使用裝置端模型:

    FIRVision *vision = [FIRVision vision];
    FIRVisionTextRecognizer *textRecognizer = [vision onDeviceTextRecognizer];

    如要使用雲端模型:

    FIRVision *vision = [FIRVision vision];
    FIRVisionTextRecognizer *textRecognizer = [vision cloudTextRecognizer];
    
    // Or, to provide language hints to assist with language detection:
    // See https://cloud.google.com/vision/docs/languages for supported languages
    FIRVisionCloudTextRecognizerOptions *options =
            [[FIRVisionCloudTextRecognizerOptions alloc] init];
    options.languageHints = @[@"en", @"hi"];
    FIRVisionTextRecognizer *textRecognizer = [vision cloudTextRecognizerWithOptions:options];
  2. 使用 UIImageVisionImage CMSampleBufferRef

    如何使用 UIImage

    1. 視需要旋轉圖片,使其 imageOrientation 屬性為 .up
    2. 使用正確旋轉的做法建立 VisionImage 物件 UIImage。請勿指定任何輪替中繼資料 (預設值) 值 (.topLeft),則必須使用。

      Swift

      let image = VisionImage(image: uiImage)

      Objective-C

      FIRVisionImage *image = [[FIRVisionImage alloc] initWithImage:uiImage];

    如何使用 CMSampleBufferRef

    1. 建立 VisionImageMetadata 物件,以指定 包含的圖片資料方向 CMSampleBufferRef 緩衝區。

      如何取得圖片方向:

      Swift

      func imageOrientation(
          deviceOrientation: UIDeviceOrientation,
          cameraPosition: AVCaptureDevice.Position
          ) -> VisionDetectorImageOrientation {
          switch deviceOrientation {
          case .portrait:
              return cameraPosition == .front ? .leftTop : .rightTop
          case .landscapeLeft:
              return cameraPosition == .front ? .bottomLeft : .topLeft
          case .portraitUpsideDown:
              return cameraPosition == .front ? .rightBottom : .leftBottom
          case .landscapeRight:
              return cameraPosition == .front ? .topRight : .bottomRight
          case .faceDown, .faceUp, .unknown:
              return .leftTop
          }
      }

      Objective-C

      - (FIRVisionDetectorImageOrientation)
          imageOrientationFromDeviceOrientation:(UIDeviceOrientation)deviceOrientation
                                 cameraPosition:(AVCaptureDevicePosition)cameraPosition {
        switch (deviceOrientation) {
          case UIDeviceOrientationPortrait:
            if (cameraPosition == AVCaptureDevicePositionFront) {
              return FIRVisionDetectorImageOrientationLeftTop;
            } else {
              return FIRVisionDetectorImageOrientationRightTop;
            }
          case UIDeviceOrientationLandscapeLeft:
            if (cameraPosition == AVCaptureDevicePositionFront) {
              return FIRVisionDetectorImageOrientationBottomLeft;
            } else {
              return FIRVisionDetectorImageOrientationTopLeft;
            }
          case UIDeviceOrientationPortraitUpsideDown:
            if (cameraPosition == AVCaptureDevicePositionFront) {
              return FIRVisionDetectorImageOrientationRightBottom;
            } else {
              return FIRVisionDetectorImageOrientationLeftBottom;
            }
          case UIDeviceOrientationLandscapeRight:
            if (cameraPosition == AVCaptureDevicePositionFront) {
              return FIRVisionDetectorImageOrientationTopRight;
            } else {
              return FIRVisionDetectorImageOrientationBottomRight;
            }
          default:
            return FIRVisionDetectorImageOrientationTopLeft;
        }
      }

      然後,建立中繼資料物件:

      Swift

      let cameraPosition = AVCaptureDevice.Position.back  // Set to the capture device you used.
      let metadata = VisionImageMetadata()
      metadata.orientation = imageOrientation(
          deviceOrientation: UIDevice.current.orientation,
          cameraPosition: cameraPosition
      )

      Objective-C

      FIRVisionImageMetadata *metadata = [[FIRVisionImageMetadata alloc] init];
      AVCaptureDevicePosition cameraPosition =
          AVCaptureDevicePositionBack;  // Set to the capture device you used.
      metadata.orientation =
          [self imageOrientationFromDeviceOrientation:UIDevice.currentDevice.orientation
                                       cameraPosition:cameraPosition];
    2. 請使用VisionImage CMSampleBufferRef 物件和輪替中繼資料:

      Swift

      let image = VisionImage(buffer: sampleBuffer)
      image.metadata = metadata

      Objective-C

      FIRVisionImage *image = [[FIRVisionImage alloc] initWithBuffer:sampleBuffer];
      image.metadata = metadata;
  3. 接著,將圖片傳遞至 process(_:completion:) 方法:

    Swift

    textRecognizer.process(visionImage) { result, error in
      guard error == nil, let result = result else {
        // ...
        return
      }
    
      // Recognized text
    }

    Objective-C

    [textRecognizer processImage:image
                      completion:^(FIRVisionText *_Nullable result,
                                   NSError *_Nullable error) {
      if (error != nil || result == nil) {
        // ...
        return;
      }
    
      // Recognized text
    }];

2. 從已辨識的文字區塊擷取文字

如果文字辨識作業成功,系統會傳回 [`VisionText`][VisionText] 物件。包含完整文字的 `VisionText` 物件 在圖像中辨識,而且零或多個 [`VisionTextBlock`][VisionTextBlock] 如需儲存大量結構化物件 建議使用 Cloud Bigtable 每個 `VisionTextBlock` 都代表矩形文字區塊,其中含有 零或多個 [`VisionTextLine`][VisionTextLine] 物件。每個「VisionTextLine」 物件包含零個或多個 [`VisionTextElement`][VisionTextElement] 物件, ,用來表示字詞和類似文字的實體 (日期、數字等)。 針對每個 `VisionTextBlock`、`VisionTextLine` 和 `VisionTextElement` 物件: 即可讓系統辨識該區域中的文字和 區域。 例如:

Swift

let resultText = result.text
for block in result.blocks {
    let blockText = block.text
    let blockConfidence = block.confidence
    let blockLanguages = block.recognizedLanguages
    let blockCornerPoints = block.cornerPoints
    let blockFrame = block.frame
    for line in block.lines {
        let lineText = line.text
        let lineConfidence = line.confidence
        let lineLanguages = line.recognizedLanguages
        let lineCornerPoints = line.cornerPoints
        let lineFrame = line.frame
        for element in line.elements {
            let elementText = element.text
            let elementConfidence = element.confidence
            let elementLanguages = element.recognizedLanguages
            let elementCornerPoints = element.cornerPoints
            let elementFrame = element.frame
        }
    }
}

Objective-C

NSString *resultText = result.text;
for (FIRVisionTextBlock *block in result.blocks) {
  NSString *blockText = block.text;
  NSNumber *blockConfidence = block.confidence;
  NSArray<FIRVisionTextRecognizedLanguage *> *blockLanguages = block.recognizedLanguages;
  NSArray<NSValue *> *blockCornerPoints = block.cornerPoints;
  CGRect blockFrame = block.frame;
  for (FIRVisionTextLine *line in block.lines) {
    NSString *lineText = line.text;
    NSNumber *lineConfidence = line.confidence;
    NSArray<FIRVisionTextRecognizedLanguage *> *lineLanguages = line.recognizedLanguages;
    NSArray<NSValue *> *lineCornerPoints = line.cornerPoints;
    CGRect lineFrame = line.frame;
    for (FIRVisionTextElement *element in line.elements) {
      NSString *elementText = element.text;
      NSNumber *elementConfidence = element.confidence;
      NSArray<FIRVisionTextRecognizedLanguage *> *elementLanguages = element.recognizedLanguages;
      NSArray<NSValue *> *elementCornerPoints = element.cornerPoints;
      CGRect elementFrame = element.frame;
    }
  }
}

即時效能改善訣竅

希望透過裝置上的模型即時辨識文字 請遵循下列準則,以達到最佳影格速率:

  • 限制對文字辨識工具的呼叫。如果新的影片影格 可在文字辨識工具執行期間捨棄外框。
  • 使用文字辨識器的輸出內容,將圖像重疊在 先從 ML Kit 取得結果,然後算繪圖片 並疊加單一步驟這麼一來,您的應用程式就會算繪到顯示途徑 每個輸入影格只能建立一次請參閱 previewOverlayViewFIRDetectionOverlayView 例如,在展示範例應用程式中使用類別。
  • 建議以較低的解析度拍攝圖片。請特別注意 這個 API 的圖片尺寸規定

後續步驟


辨識文件圖片中的文字

如要辨識文件中的文字,請設定並執行雲端式 與文件文字辨識工具搭配使用

以下說明文件文字辨識 API 提供的介面 是為了方便處理文件圖片。不過 如果您偏好稀疏文字 API 提供的介面,則可以使用這個 API 只要將 Cloud 文字辨識工具設為 使用密集文字模型

如何使用文件文字辨識 API:

1. 執行文字辨識工具

將圖片做為 UIImageCMSampleBufferRef 傳遞至 VisionDocumentTextRecognizerprocess(_:completion:) 方法:

  1. 呼叫即可取得 VisionDocumentTextRecognizer 的例項 cloudDocumentTextRecognizer:

    Swift

    let vision = Vision.vision()
    let textRecognizer = vision.cloudDocumentTextRecognizer()
    
    // Or, to provide language hints to assist with language detection:
    // See https://cloud.google.com/vision/docs/languages for supported languages
    let options = VisionCloudDocumentTextRecognizerOptions()
    options.languageHints = ["en", "hi"]
    let textRecognizer = vision.cloudDocumentTextRecognizer(options: options)

    Objective-C

    FIRVision *vision = [FIRVision vision];
    FIRVisionDocumentTextRecognizer *textRecognizer = [vision cloudDocumentTextRecognizer];
    
    // Or, to provide language hints to assist with language detection:
    // See https://cloud.google.com/vision/docs/languages for supported languages
    FIRVisionCloudDocumentTextRecognizerOptions *options =
            [[FIRVisionCloudDocumentTextRecognizerOptions alloc] init];
    options.languageHints = @[@"en", @"hi"];
    FIRVisionDocumentTextRecognizer *textRecognizer = [vision cloudDocumentTextRecognizerWithOptions:options];
  2. 使用 UIImageVisionImage CMSampleBufferRef

    如何使用 UIImage

    1. 視需要旋轉圖片,使其 imageOrientation 屬性為 .up
    2. 使用正確旋轉的做法建立 VisionImage 物件 UIImage。請勿指定任何輪替中繼資料 (預設值) 值 (.topLeft),則必須使用。

      Swift

      let image = VisionImage(image: uiImage)

      Objective-C

      FIRVisionImage *image = [[FIRVisionImage alloc] initWithImage:uiImage];

    如何使用 CMSampleBufferRef

    1. 建立 VisionImageMetadata 物件,以指定 包含的圖片資料方向 CMSampleBufferRef 緩衝區。

      如何取得圖片方向:

      Swift

      func imageOrientation(
          deviceOrientation: UIDeviceOrientation,
          cameraPosition: AVCaptureDevice.Position
          ) -> VisionDetectorImageOrientation {
          switch deviceOrientation {
          case .portrait:
              return cameraPosition == .front ? .leftTop : .rightTop
          case .landscapeLeft:
              return cameraPosition == .front ? .bottomLeft : .topLeft
          case .portraitUpsideDown:
              return cameraPosition == .front ? .rightBottom : .leftBottom
          case .landscapeRight:
              return cameraPosition == .front ? .topRight : .bottomRight
          case .faceDown, .faceUp, .unknown:
              return .leftTop
          }
      }

      Objective-C

      - (FIRVisionDetectorImageOrientation)
          imageOrientationFromDeviceOrientation:(UIDeviceOrientation)deviceOrientation
                                 cameraPosition:(AVCaptureDevicePosition)cameraPosition {
        switch (deviceOrientation) {
          case UIDeviceOrientationPortrait:
            if (cameraPosition == AVCaptureDevicePositionFront) {
              return FIRVisionDetectorImageOrientationLeftTop;
            } else {
              return FIRVisionDetectorImageOrientationRightTop;
            }
          case UIDeviceOrientationLandscapeLeft:
            if (cameraPosition == AVCaptureDevicePositionFront) {
              return FIRVisionDetectorImageOrientationBottomLeft;
            } else {
              return FIRVisionDetectorImageOrientationTopLeft;
            }
          case UIDeviceOrientationPortraitUpsideDown:
            if (cameraPosition == AVCaptureDevicePositionFront) {
              return FIRVisionDetectorImageOrientationRightBottom;
            } else {
              return FIRVisionDetectorImageOrientationLeftBottom;
            }
          case UIDeviceOrientationLandscapeRight:
            if (cameraPosition == AVCaptureDevicePositionFront) {
              return FIRVisionDetectorImageOrientationTopRight;
            } else {
              return FIRVisionDetectorImageOrientationBottomRight;
            }
          default:
            return FIRVisionDetectorImageOrientationTopLeft;
        }
      }

      然後,建立中繼資料物件:

      Swift

      let cameraPosition = AVCaptureDevice.Position.back  // Set to the capture device you used.
      let metadata = VisionImageMetadata()
      metadata.orientation = imageOrientation(
          deviceOrientation: UIDevice.current.orientation,
          cameraPosition: cameraPosition
      )

      Objective-C

      FIRVisionImageMetadata *metadata = [[FIRVisionImageMetadata alloc] init];
      AVCaptureDevicePosition cameraPosition =
          AVCaptureDevicePositionBack;  // Set to the capture device you used.
      metadata.orientation =
          [self imageOrientationFromDeviceOrientation:UIDevice.currentDevice.orientation
                                       cameraPosition:cameraPosition];
    2. 請使用VisionImage CMSampleBufferRef 物件和輪替中繼資料:

      Swift

      let image = VisionImage(buffer: sampleBuffer)
      image.metadata = metadata

      Objective-C

      FIRVisionImage *image = [[FIRVisionImage alloc] initWithBuffer:sampleBuffer];
      image.metadata = metadata;
  3. 接著,將圖片傳遞至 process(_:completion:) 方法:

    Swift

    textRecognizer.process(visionImage) { result, error in
      guard error == nil, let result = result else {
        // ...
        return
      }
    
      // Recognized text
    }

    Objective-C

    [textRecognizer processImage:image
                      completion:^(FIRVisionDocumentText *_Nullable result,
                                   NSError *_Nullable error) {
      if (error != nil || result == nil) {
        // ...
        return;
      }
    
        // Recognized text
    }];

2. 從已辨識的文字區塊擷取文字

如果文字辨識作業成功,系統會傳回 VisionDocumentText 物件。VisionDocumentText 物件 包含圖片中可辨識的完整文字及物件階層 反映公認文件的結構:

VisionDocumentTextBlockVisionDocumentTextParagraphVisionDocumentTextWordVisionDocumentTextSymbol 物件,您可以取得 可在區域辨識的文字和區域的邊界座標。

例如:

Swift

let resultText = result.text
for block in result.blocks {
    let blockText = block.text
    let blockConfidence = block.confidence
    let blockRecognizedLanguages = block.recognizedLanguages
    let blockBreak = block.recognizedBreak
    let blockCornerPoints = block.cornerPoints
    let blockFrame = block.frame
    for paragraph in block.paragraphs {
        let paragraphText = paragraph.text
        let paragraphConfidence = paragraph.confidence
        let paragraphRecognizedLanguages = paragraph.recognizedLanguages
        let paragraphBreak = paragraph.recognizedBreak
        let paragraphCornerPoints = paragraph.cornerPoints
        let paragraphFrame = paragraph.frame
        for word in paragraph.words {
            let wordText = word.text
            let wordConfidence = word.confidence
            let wordRecognizedLanguages = word.recognizedLanguages
            let wordBreak = word.recognizedBreak
            let wordCornerPoints = word.cornerPoints
            let wordFrame = word.frame
            for symbol in word.symbols {
                let symbolText = symbol.text
                let symbolConfidence = symbol.confidence
                let symbolRecognizedLanguages = symbol.recognizedLanguages
                let symbolBreak = symbol.recognizedBreak
                let symbolCornerPoints = symbol.cornerPoints
                let symbolFrame = symbol.frame
            }
        }
    }
}

Objective-C

NSString *resultText = result.text;
for (FIRVisionDocumentTextBlock *block in result.blocks) {
  NSString *blockText = block.text;
  NSNumber *blockConfidence = block.confidence;
  NSArray<FIRVisionTextRecognizedLanguage *> *blockRecognizedLanguages = block.recognizedLanguages;
  FIRVisionTextRecognizedBreak *blockBreak = block.recognizedBreak;
  CGRect blockFrame = block.frame;
  for (FIRVisionDocumentTextParagraph *paragraph in block.paragraphs) {
    NSString *paragraphText = paragraph.text;
    NSNumber *paragraphConfidence = paragraph.confidence;
    NSArray<FIRVisionTextRecognizedLanguage *> *paragraphRecognizedLanguages = paragraph.recognizedLanguages;
    FIRVisionTextRecognizedBreak *paragraphBreak = paragraph.recognizedBreak;
    CGRect paragraphFrame = paragraph.frame;
    for (FIRVisionDocumentTextWord *word in paragraph.words) {
      NSString *wordText = word.text;
      NSNumber *wordConfidence = word.confidence;
      NSArray<FIRVisionTextRecognizedLanguage *> *wordRecognizedLanguages = word.recognizedLanguages;
      FIRVisionTextRecognizedBreak *wordBreak = word.recognizedBreak;
      CGRect wordFrame = word.frame;
      for (FIRVisionDocumentTextSymbol *symbol in word.symbols) {
        NSString *symbolText = symbol.text;
        NSNumber *symbolConfidence = symbol.confidence;
        NSArray<FIRVisionTextRecognizedLanguage *> *symbolRecognizedLanguages = symbol.recognizedLanguages;
        FIRVisionTextRecognizedBreak *symbolBreak = symbol.recognizedBreak;
        CGRect symbolFrame = symbol.frame;
      }
    }
  }
}

後續步驟