使用 Firebase ML 识别图片中的文本 (iOS)

您可以使用 Firebase ML 识别图片中的文本。Firebase ML 既有可用于识别图片中的文本(例如街道标志的文本)的通用 API,也有针对识别文档文本而优化的 API。

准备工作

    如果您尚未将 Firebase 添加到自己的应用中,请按照入门指南中的步骤进行添加。

    使用 Swift Package Manager 安装和管理 Firebase 依赖项。

    1. 在 Xcode 中打开您的应用项目,依次点击 File(文件)> Add Packages(添加软件包)
    2. 出现提示时,添加 Firebase Apple 平台 SDK 代码库:
    3.   https://github.com/firebase/firebase-ios-sdk.git
    4. 选择 Firebase ML 库。
    5. -ObjC 标志添加到目标 build 设置的“其他链接器标志”部分。
    6. 完成之后,Xcode 将会自动开始在后台解析和下载您的依赖项。

    接下来,执行一些应用内设置:

    1. 在您的应用中导入 Firebase:

      Swift

      import FirebaseMLModelDownloader

      Objective-C

      @import FirebaseMLModelDownloader;
  1. 如果您尚未为项目启用云端 API,请立即完成以下操作:

    1. 打开 Firebase 控制台的 Firebase ML API 页面
    2. 如果您尚未将项目升级到 Blaze 定价方案,请点击升级以执行此操作。(只有在您的项目未采用 Blaze 方案时,系统才会提示您进行升级。)

      只有 Blaze 级项目才能使用基于 Cloud 的 API。

    3. 如果尚未启用基于 Cloud 的 API,请点击启用基于 Cloud 的 API

现在,您可以开始识别图片中的文本了。

输入图片指南

  • 为了使 Firebase ML 准确识别文本,输入图片必须包含由足够像素数据表示的文本。理想情况下,对于拉丁文本,每个字符应至少为 16x16 像素。对于中文、日文和韩文文本,每个字符应为 24x24 像素。对于所有语言,字符像素大于 24x24 通常不会增加准确性。

    例如,640x480 像素的图片可能非常适合用于扫描占据图片整个宽度的名片。如需扫描打印在信纸大小纸张上的文档,可能需要 720x1280 像素的图片。

  • 图片聚焦不佳会影响文本识别的准确性。如果您无法获得满意的结果,请尝试让用户重新采集图片。


识别图片中的文本

如需识别图片中的文本,请按照以下说明运行文本识别器。

1. 运行文本识别器

将图片作为 UIImageCMSampleBufferRef 传递给 VisionTextRecognizerprocess(_:completion:) 方法:

  1. 通过调用 cloudTextRecognizer 获取 VisionTextRecognizer 的实例:

    Swift

    let vision = Vision.vision()
    let textRecognizer = vision.cloudTextRecognizer()
    
    // Or, to provide language hints to assist with language detection:
    // See https://cloud.google.com/vision/docs/languages for supported languages
    let options = VisionCloudTextRecognizerOptions()
    options.languageHints = ["en", "hi"]
    let textRecognizer = vision.cloudTextRecognizer(options: options)

    Objective-C

    FIRVision *vision = [FIRVision vision];
    FIRVisionTextRecognizer *textRecognizer = [vision cloudTextRecognizer];
    
    // Or, to provide language hints to assist with language detection:
    // See https://cloud.google.com/vision/docs/languages for supported languages
    FIRVisionCloudTextRecognizerOptions *options =
            [[FIRVisionCloudTextRecognizerOptions alloc] init];
    options.languageHints = @[@"en", @"hi"];
    FIRVisionTextRecognizer *textRecognizer = [vision cloudTextRecognizerWithOptions:options];
  2. 如需调用 Cloud Vision,图片的格式必须为 base64 编码字符串。如需处理 UIImage,请执行以下操作:

    Swift

    guard let imageData = uiImage.jpegData(compressionQuality: 1.0) else { return }
    let base64encodedImage = imageData.base64EncodedString()

    Objective-C

    NSData *imageData = UIImageJPEGRepresentation(uiImage, 1.0f);
    NSString *base64encodedImage =
      [imageData base64EncodedStringWithOptions:NSDataBase64Encoding76CharacterLineLength];
  3. 然后,将图片传递给 process(_:completion:) 方法:

    Swift

    textRecognizer.process(visionImage) { result, error in
      guard error == nil, let result = result else {
        // ...
        return
      }
    
      // Recognized text
    }

    Objective-C

    [textRecognizer processImage:image
                      completion:^(FIRVisionText *_Nullable result,
                                   NSError *_Nullable error) {
      if (error != nil || result == nil) {
        // ...
        return;
      }
    
      // Recognized text
    }];

2. 从识别出的文本块中提取文本

如果文本识别操作成功,它将返回一个 VisionText 对象。VisionText对象包含图片中识别到的完整文本以及零个或零个以上的 VisionTextBlock 对象。

每个 VisionTextBlock 表示一个矩形文本块,其中包含零个或零个以上的 VisionTextLine 对象。每个 VisionTextLine 对象包含零个或零个以上的 VisionTextElement 对象,这些对象表示字词和类似字词的实体(日期、数字等)。

对于每个 VisionTextBlockVisionTextLineVisionTextElement 对象,您可以获取区域中识别出的文本以及该区域的边界坐标。

例如:

Swift

let resultText = result.text
for block in result.blocks {
    let blockText = block.text
    let blockConfidence = block.confidence
    let blockLanguages = block.recognizedLanguages
    let blockCornerPoints = block.cornerPoints
    let blockFrame = block.frame
    for line in block.lines {
        let lineText = line.text
        let lineConfidence = line.confidence
        let lineLanguages = line.recognizedLanguages
        let lineCornerPoints = line.cornerPoints
        let lineFrame = line.frame
        for element in line.elements {
            let elementText = element.text
            let elementConfidence = element.confidence
            let elementLanguages = element.recognizedLanguages
            let elementCornerPoints = element.cornerPoints
            let elementFrame = element.frame
        }
    }
}

Objective-C

NSString *resultText = result.text;
for (FIRVisionTextBlock *block in result.blocks) {
  NSString *blockText = block.text;
  NSNumber *blockConfidence = block.confidence;
  NSArray<FIRVisionTextRecognizedLanguage *> *blockLanguages = block.recognizedLanguages;
  NSArray<NSValue *> *blockCornerPoints = block.cornerPoints;
  CGRect blockFrame = block.frame;
  for (FIRVisionTextLine *line in block.lines) {
    NSString *lineText = line.text;
    NSNumber *lineConfidence = line.confidence;
    NSArray<FIRVisionTextRecognizedLanguage *> *lineLanguages = line.recognizedLanguages;
    NSArray<NSValue *> *lineCornerPoints = line.cornerPoints;
    CGRect lineFrame = line.frame;
    for (FIRVisionTextElement *element in line.elements) {
      NSString *elementText = element.text;
      NSNumber *elementConfidence = element.confidence;
      NSArray<FIRVisionTextRecognizedLanguage *> *elementLanguages = element.recognizedLanguages;
      NSArray<NSValue *> *elementCornerPoints = element.cornerPoints;
      CGRect elementFrame = element.frame;
    }
  }
}

后续步骤


识别文档图片中的文本

如需识别文档的文本,请按照以下说明配置并运行文档文本识别器。

下文所述的文档文本识别 API 提供了一个旨在更方便地处理文档图片的接口。但是,如果您更喜欢使用稀疏文本 API 提供的接口,则可以改用该接口来扫描文档(只需将云端文本识别器配置为使用密集文本模型即可)。

如需使用文档文本识别 API,请执行以下操作:

1. 运行文本识别器

将图片作为 UIImageCMSampleBufferRef 传递给 VisionDocumentTextRecognizerprocess(_:completion:) 方法:

  1. 通过调用 cloudDocumentTextRecognizer 获取 VisionDocumentTextRecognizer 的实例:

    Swift

    let vision = Vision.vision()
    let textRecognizer = vision.cloudDocumentTextRecognizer()
    
    // Or, to provide language hints to assist with language detection:
    // See https://cloud.google.com/vision/docs/languages for supported languages
    let options = VisionCloudDocumentTextRecognizerOptions()
    options.languageHints = ["en", "hi"]
    let textRecognizer = vision.cloudDocumentTextRecognizer(options: options)

    Objective-C

    FIRVision *vision = [FIRVision vision];
    FIRVisionDocumentTextRecognizer *textRecognizer = [vision cloudDocumentTextRecognizer];
    
    // Or, to provide language hints to assist with language detection:
    // See https://cloud.google.com/vision/docs/languages for supported languages
    FIRVisionCloudDocumentTextRecognizerOptions *options =
            [[FIRVisionCloudDocumentTextRecognizerOptions alloc] init];
    options.languageHints = @[@"en", @"hi"];
    FIRVisionDocumentTextRecognizer *textRecognizer = [vision cloudDocumentTextRecognizerWithOptions:options];
  2. 如需调用 Cloud Vision,图片的格式必须为 base64 编码字符串。如需处理 UIImage,请执行以下操作:

    Swift

    guard let imageData = uiImage.jpegData(compressionQuality: 1.0) else { return }
    let base64encodedImage = imageData.base64EncodedString()

    Objective-C

    NSData *imageData = UIImageJPEGRepresentation(uiImage, 1.0f);
    NSString *base64encodedImage =
      [imageData base64EncodedStringWithOptions:NSDataBase64Encoding76CharacterLineLength];
  3. 然后,将图片传递给 process(_:completion:) 方法:

    Swift

    textRecognizer.process(visionImage) { result, error in
      guard error == nil, let result = result else {
        // ...
        return
      }
    
      // Recognized text
    }

    Objective-C

    [textRecognizer processImage:image
                      completion:^(FIRVisionDocumentText *_Nullable result,
                                   NSError *_Nullable error) {
      if (error != nil || result == nil) {
        // ...
        return;
      }
    
        // Recognized text
    }];

2. 从识别出的文本块中提取文本

如果文本识别操作成功,它将返回一个 VisionDocumentText 对象。VisionDocumentText 对象包含图片中识别到的完整文本以及反映所识别的文档结构的对象层次结构:

对于每个 VisionDocumentTextBlockVisionDocumentTextParagraphVisionDocumentTextWordVisionDocumentTextSymbol 对象,您可以获取区域中识别出的文本以及该区域的边界坐标。

例如:

Swift

let resultText = result.text
for block in result.blocks {
    let blockText = block.text
    let blockConfidence = block.confidence
    let blockRecognizedLanguages = block.recognizedLanguages
    let blockBreak = block.recognizedBreak
    let blockCornerPoints = block.cornerPoints
    let blockFrame = block.frame
    for paragraph in block.paragraphs {
        let paragraphText = paragraph.text
        let paragraphConfidence = paragraph.confidence
        let paragraphRecognizedLanguages = paragraph.recognizedLanguages
        let paragraphBreak = paragraph.recognizedBreak
        let paragraphCornerPoints = paragraph.cornerPoints
        let paragraphFrame = paragraph.frame
        for word in paragraph.words {
            let wordText = word.text
            let wordConfidence = word.confidence
            let wordRecognizedLanguages = word.recognizedLanguages
            let wordBreak = word.recognizedBreak
            let wordCornerPoints = word.cornerPoints
            let wordFrame = word.frame
            for symbol in word.symbols {
                let symbolText = symbol.text
                let symbolConfidence = symbol.confidence
                let symbolRecognizedLanguages = symbol.recognizedLanguages
                let symbolBreak = symbol.recognizedBreak
                let symbolCornerPoints = symbol.cornerPoints
                let symbolFrame = symbol.frame
            }
        }
    }
}

Objective-C

NSString *resultText = result.text;
for (FIRVisionDocumentTextBlock *block in result.blocks) {
  NSString *blockText = block.text;
  NSNumber *blockConfidence = block.confidence;
  NSArray<FIRVisionTextRecognizedLanguage *> *blockRecognizedLanguages = block.recognizedLanguages;
  FIRVisionTextRecognizedBreak *blockBreak = block.recognizedBreak;
  CGRect blockFrame = block.frame;
  for (FIRVisionDocumentTextParagraph *paragraph in block.paragraphs) {
    NSString *paragraphText = paragraph.text;
    NSNumber *paragraphConfidence = paragraph.confidence;
    NSArray<FIRVisionTextRecognizedLanguage *> *paragraphRecognizedLanguages = paragraph.recognizedLanguages;
    FIRVisionTextRecognizedBreak *paragraphBreak = paragraph.recognizedBreak;
    CGRect paragraphFrame = paragraph.frame;
    for (FIRVisionDocumentTextWord *word in paragraph.words) {
      NSString *wordText = word.text;
      NSNumber *wordConfidence = word.confidence;
      NSArray<FIRVisionTextRecognizedLanguage *> *wordRecognizedLanguages = word.recognizedLanguages;
      FIRVisionTextRecognizedBreak *wordBreak = word.recognizedBreak;
      CGRect wordFrame = word.frame;
      for (FIRVisionDocumentTextSymbol *symbol in word.symbols) {
        NSString *symbolText = symbol.text;
        NSNumber *symbolConfidence = symbol.confidence;
        NSArray<FIRVisionTextRecognizedLanguage *> *symbolRecognizedLanguages = symbol.recognizedLanguages;
        FIRVisionTextRecognizedBreak *symbolBreak = symbol.recognizedBreak;
        CGRect symbolFrame = symbol.frame;
      }
    }
  }
}

后续步骤