Knowledge Extraction Recipes - Forms

Retrieving information from documents and forms has long been a challenge, and even now at the time of writing,organisations are still handling significant amounts of paper forms that need to be scanned, classified and minedfor specific information to enable downstream automation and efficiencies. Automating this extraction and applyingintelligence is in fact a fundamental step toward digital transformation that organisations are still strugglingto solve in an efficient and scalable manner.

An example could be a bank that receives hundreds of kilograms of very diverse remittance forms a day that need to beprocessed manually by people in order to extract a few key fields. Or medicinal prescriptions need to be automated to extract theprescribed medication and quantity.

Typically organisations will have built text mining and search solutions which are often tailored for a scenario,with baked in application logic, resulting in an often brittle solution that is difficult and expensive to maintain.

Thanks to the breakthroughs and rapid innovation in the machine learning fields of Computer Vision and NaturalLanguage Processing (NLP), reliable options are now available to provide data driven solutions that generaliseand provide high degrees of accuracy in extracting information from structured forms.

Coupled with Azure services this provides rapidly deployable, cost efficient andscalable solutions ready for production workloads.

Overview

The goal of this Playbook is to build a set of guidance, tools, examples and documentation that illustrate someknown techniques for information extraction, all of which have been applied in real customer solutions.

We hope that the Playbook can significantly reduce the overall development time by simplifying the decision makingprocess from defining the business problem to analysis and development.

The first focus of the Playbook is extraction of information from Forms.

Intended audience

The intended audience of this Playbook include:

• Engineering/project leads
• Data scientists/data engineers
• Machine learning engineers
• Software engineers

How this Playbook is structured

This Playbook aims to provide step-by-step guidance for each phase of a typical Forms Extraction project alongside typical considerations, key outcomes and code accelerators per phase. To follow the guidance process see the Walkthrough or dip into the individual code accelerators

Getting Started

The best place to start if this is your first foray into this Playbook is with the Checklist, and then the Walkthrough to ensure that the most importantpoints are addressed in order to build a successful solution in this space.

Terminology used in this Playbook

We refer to the Supervised version of the Form Recognizer service when the argument Use Labels set to True when training, and the Unsupervised version of Form Recognizer as when the argument Use Labels is set to False.

We refer to a form issuer as being the unique source of a form, for example, the vendor of an invoice, or the bank of origin of an application form.

End to end demos ✨

Code accelerators

The following code accelerators serve as starting points to try approaches that are known to work for KnowledgeExtraction.Note - these accelerators need to be adapted to your data and tested and profiled, they are not production readyand need to be incorporated into your pipeline and profiled

The code accelerators included are available in Jupyter notebooks, APIs and python scripts that showcase some of the scenariosin this repository using diverse approaches.

PowerApps ✨

NEW (▀̿Ĺ̯▀̿ ̿)

Example Pipelines

The Pipelines section contains some example patterns and pipelines for Knowledge Extraction using Azure Services.

Tips and Best Practices for Form Recognizer

For tips and best practices for managing Form Recognizer models via MLOps and deployment pipelines, view MLOps Tips and Tricks for Form Recognizer.

Example Scenarios

This section contains some documented common scenarios

Azure Services used in this repository

Azure Computer Vision OCR

Read APIdetects text content in an image using our latest recognition models and converts the identified text into amachine-readable character stream. It's optimized for text-heavy images (such as documents that have been digitallyscanned) and for images with a lot of visual noise. It will determine which recognition model to use for each lineof text, supporting images with both printed and handwritten text. The Read API executes asynchronously becauselarger documents can take several minutes to return a result.

OCR API Computer Vision's optical character recognition (OCR)API is similar to the Read API, but it executes synchronously and is not optimized for large documents. It uses anearlier recognition model but works with more languages

Azure Cognitive Search

Azure Cognitive Search is a fully managed search as a service to reducecomplexity and scale easily including:

• Auto-complete, geospatial search, filtering, and faceting capabilities for a rich user experience
• Built-in AI capabilities including OCR, key phrase extraction, and named entity recognition to unlock insights
• Flexible integration of custom models, classifiers, and rankers to fit your domain-specific needs

Form Recognizer Service

Form Recognizer applies advancedmachine learning to accurately extract text, key/value pairs and tables from documents.

The Form Recognizer has two modesof operation:

• Custom Model:This mode can be trained to recognise specific form types based on your own example data set
• Prebuilt Receipt Model: This model is pre-trained (requires no training from you) to reocgnise and extract key data points from receipts(i.e. till receipts, resturant bills, general retail receipts etc)

The Custom Model requires the following for training:

• Format must be JPG, PNG, or PDF (text or scanned). Text-embedded PDFs are best because there's no possibility of error in character extraction and location.
• If your PDFs are password-locked, you must remove the lock before submitting them.
• File size must be less than 4 MB.
• For images, dimensions must be between 600 x 100 pixels and 4200 x 4200 pixels.
• If scanned from paper documents, forms should be high-quality scans.
• Text must use the Latin alphabet (English characters).
• Data must contain keys and values.
• Keys can appear above or to the left of the values, but not below or to the right.

Form Recognizer doesn't currently support these types of input data:

• Complex tables (nested tables, merged headers or cells, and so on).
• Checkboxes or radio buttons.
• PDF documents longer than 50 pages.

See more on training Form Recognizer here

The requirements for the prebuilt receipt model are slightly different.

• Format must be JPEG, PNG, BMP, PDF (text or scanned) or TIFF.
• File size must be less than 20 MB.
• Image dimensions must be between 50 x 50 pixels and 10000 x 10000 pixels.
• PDF dimensions must be at most 17 x 17 inches, corresponding to Legal or A3 paper sizes and smaller.
• For PDF and TIFF, only the first 200 pages are processed (with a free tier subscription, only the first two pagesare processed).

Azure Machine Learning service

Azure Machine Learning service is a cloudservice used to train, deploy, automate, and manage machine learning models, all at the broad scale that the cloudprovides. AzureML is presented in notebooks across different scenarios to enhance the efficiency of developing NaturalLanguage systems at scale and for various AI model development related tasks like:

• Accessing Datastoresto easily read and write your data in Azure storage services such as blob storage or file share.
• Scaling up and out on Azure Machine Learning Compute.
• Automated Machine Learning which builds high quality machine learning models by automating model and hyperparameter selection.
• Tracking experiments and monitoring metrics to enhance the model creation process.
• Distributed Training
• Hyperparameter tuning
• Deploying the trained machine learning model as a web service to Azure Container Instance for deveopment and test, or for low scale, CPU-based workloads.
• Deploying the trained machine learning model as a web service to Azure Kubernetes Service for high-scale production deployments and provides autoscaling, and fast response times.

To successfully run these code accelerators, you will need an Azure subscriptionor can try Azure for free. There may be other Azure services or productsused in the code. Introduction and/or reference of those will be provided in the code themselves.

Contributors ✨

Alessandro Jannuzzi	Alex Hocking	Ayaka Hara	Dariusz Parys	Ibrahim Kivanc	Karol Zak	Katia Gil Guzman
Krit Kamtuo	Marcia Dos Santos	Nuno Silva	Ignacio Floristan	Oleksiy Shepetko	Omri Mendels	Pretesh Patel
Raj Nemani	Sergii Baidachnyi	Tomi Paananen	Martin Kearn	Shane Peckham	Mick Vleeshouwer	Jon Malsan
Eva Mok	Mitchell Overfield	Jafar Al-Kofahi	Daniel Fatade	Steve Pucelik

Contributing

See CONTRIBUTING.md for contribution guidelines.

Additional References

Please refer to the following fantastic references for additional material relevant to knowledge extraction:

Azure Cognitive Search additional links

• Azure Search Knowledge mining

• Azure Search Knowledge Mining Bootcamp

• Azure Search custom skills

General and custom information extraction

• Custom NLP models - Microsoft NLP Recipes
• Spacy Phrasematcher
• RegEx debugger and code snippets
• Recognizers-Text (entity recognition for numbers, units, and date/time; in multiple languages)

Addresses

• Address parsing

Image pre-processing

• Compressing and enhancing hand-written notes

PDF conversion

• PDF Miner
• PDF Focus
• ImageMagick

OCR

• OCR Labelling Tool
• Azure OCR Visualiser
• Tesseract

MLOps

• MLOps
• MLOpsPython

Intelligent Kiosk Demo

• Intelligent Kiosk