nidaba

Expandable and scalable OCR pipeline

Project description

Overview

Nidaba is the central controller for the entire OGL OCR pipeline. It oversees and automates the process of converting raw images into citable collections of digitized texts.

It offers the following functionality:

Grayscale Conversion
Binarization utilizing [Sauvola](http://www.mediateam.oulu.fi/publications/pdf/24.p) adaptive thresholding or leptonica’s [Otsu](http://www.leptonica.com/binarization.html) thresholding with background normalization
Deskewing
Dewarping
Integration of [tesseract](http://code.google.com/p/tesseract-ocr/) and ocropus OCR engines
Merging multiple hOCR documents using scoring

As it is designed to use a common storage medium on network attached storage and the [celery](http://celeryproject.org) distributed task queue it scales nicely to multi-machine clusters.

Build

To build Nidaba run

` $ pip install . `

in the root directory or install using pypi:

` $ pip install nibada `

Some useful tasks have external dependencies. A good start is:

` # apt-get install tesseract-ocr leptonica-progs ``

Tests

Per default no dictionaries and OCR models necessary to runs the tests are installed. To download the necessary files run:

` $ python setup.py download `

` $ python setup.py test `

Tests for modules that call external programs, at the time only tesseract and ocropus, will be skipped if these aren’t installed.

Running

First edit (the installed) nidaba.yaml and celery.yaml to fit your needs. Have a look at the [docs](https:///mittagessen.github.io/nidaba) if you haven’t set up a celery-based application before.

Then start up the celery daemon with something like:

` $ celery -A nidaba worker `

Next jobs can be added to the pipeline using the nidaba executable:

` $ nidaba batch --binarize "sauvola:whsize=10;whsize=20;whsize=30;whsize=40,factor=0.6" --ocr tesseract:eng -- ./input.tiff Preparing filestore....done. Building batch...done. 25d79a54-9d4a-4939-acb6-8e168d6dbc7c `

Using the return code the current state of the job can be retrieved:

` $ nidaba status 25d79a54-9d4a-4939-acb6-8e168d6dbc7c PENDING `

When the job has been processed the status command will return a list of paths containing the final output:

` $ nidaba status 25d79a54-9d4a-4939-acb6-8e168d6dbc7c SUCCESS input.tiff -> /home/mittagessen/OCR/97150c41-82a9-4935-8063-9295a2eb2a7f/input_img.rgb_to_gray_binarize.sauvola_10_0.35_ocr.tesseract_eng.tiff.hocr input.tiff -> /home/mittagessen/OCR/97150c41-82a9-4935-8063-9295a2eb2a7f/input_img.rgb_to_gray_binarize.sauvola_20_0.35_ocr.tesseract_eng.tiff.hocr input.tiff -> /home/mittagessen/OCR/97150c41-82a9-4935-8063-9295a2eb2a7f/input_img.rgb_to_gray_binarize.sauvola_30_0.35_ocr.tesseract_eng.tiff.hocr input.tiff -> /home/mittagessen/OCR/97150c41-82a9-4935-8063-9295a2eb2a7f/input_img.rgb_to_gray_binarize.sauvola_40_0.6_ocr.tesseract_eng.tiff.hocr `

Documentation

Want to learn more? [Read the Docs](https:///openphilology.github.io/nidaba/)

Project details

Release history Release notifications | RSS feed

2.0.4

Oct 26, 2017

2.0.1

Mar 11, 2017

0.9.8

Jan 22, 2016

0.9.8.dev10 pre-release

Jan 22, 2016

0.9.7

Nov 25, 2015

0.9.6

Nov 20, 2015

0.9.5

Nov 19, 2015

0.9.4

Oct 27, 2015

0.9.3

Oct 24, 2015

0.9.2

Oct 24, 2015

0.9.1

Oct 23, 2015

0.9.0

Oct 21, 2015

0.6.1

Oct 9, 2015

0.6.0

Sep 18, 2015

0.5.4

Aug 26, 2015

0.5.3

Aug 25, 2015

0.5.2

Aug 20, 2015

0.5.1

Aug 18, 2015

0.5.0

Aug 17, 2015

0.4.2

Aug 12, 2015

0.4.1

Aug 5, 2015

0.4.0

Aug 4, 2015

0.3.18

Jun 4, 2015

0.3.17

Jun 4, 2015

0.3.17.dev19 pre-release

Jun 2, 2015

0.3.16

Jun 1, 2015

0.3.15

May 19, 2015

0.3.14

May 18, 2015

0.3.13

May 18, 2015

0.3.11

May 5, 2015

0.3.10

Apr 29, 2015

0.3.9

Apr 29, 2015

0.3.8

Apr 29, 2015

0.3.7

Apr 21, 2015

0.3.6.post1

Apr 21, 2015

0.3.6

Apr 21, 2015

0.3.5

Apr 21, 2015

This version

0.3.4

Apr 19, 2015

0.3.3

Apr 16, 2015

0.3.2

Apr 15, 2015

0.3.1

Apr 7, 2015

0.3.0

Apr 1, 2015

0.2.0

Mar 24, 2015

0.1.0.post4

Mar 9, 2015

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

nidaba-0.3.4.tar.gz (60.7 kB view details)

Uploaded Apr 19, 2015 Source

Built Distribution

nidaba-0.3.4-py2-none-any.whl (53.8 kB view details)

Uploaded Apr 19, 2015 Python 2

File details

Details for the file nidaba-0.3.4.tar.gz.

File metadata

Download URL: nidaba-0.3.4.tar.gz
Upload date: Apr 19, 2015
Size: 60.7 kB
Tags: Source
Uploaded using Trusted Publishing? No

File hashes

Hashes for nidaba-0.3.4.tar.gz
Algorithm	Hash digest
SHA256	`a0913bdb6aa2150406a251e968ac98d47920efc4aa72b63ea408a3eea033f5fb`
MD5	`ff8ca58f0d3dca400594e6bedd0dda5b`
BLAKE2b-256	`6c3dbca636efcab93b6b39a87c1043aebbf7f908bf3f6b1dec058727ce4a3178`

See more details on using hashes here.

File details

Details for the file nidaba-0.3.4-py2-none-any.whl.

File metadata

Download URL: nidaba-0.3.4-py2-none-any.whl
Upload date: Apr 19, 2015
Size: 53.8 kB
Tags: Python 2
Uploaded using Trusted Publishing? No

File hashes

Hashes for nidaba-0.3.4-py2-none-any.whl
Algorithm	Hash digest
SHA256	`804377afde9168c05239f7bdd233dbf95ad18429ab1da3e5bdd4c46d32361cb4`
MD5	`f20b26b5d9909fec1352df702846a3a2`
BLAKE2b-256	`dc85f683f801e213e68e7fc30b5e8d028e82ac92de288495b99b4d9c741e4631`