Expandable and scalable OCR pipeline
Project description
Overview
========
Nidaba is the central controller for the entire OGL OCR pipeline. It oversees and
automates the process of converting raw images into citable collections of
digitized texts. Images can be uploaded directly via Nidaba' RESTful web portal,
or can be selected from preexisting images located on Nidaba' image repository.
It offers the following functionality:
* Grayscale Conversion
* Binarization utilizing
[Sauvola](http://www.mediateam.oulu.fi/publications/pdf/24.p) adaptive
thresholding or leptonica's
[Otsu](http://www.leptonica.com/binarization.html) thresholding with
background normalization
* Deskewing
* Dewarping
* Integration of [tesseract](http://code.google.com/p/tesseract-ocr/) and ocropus OCR
engines
* Merging multiple hOCR documents using scoring
As it is designed to use a common storage medium on network attached storage
and the [celery](http://celeryproject.org) distributed task queue it scales
nicely to multi-machine clusters.
Build
=====
To build Nidaba run
```
$ pip install .
```
in the root directory. pip is required to extract the dependencies for
setuptools from the requirements file, so there is no reason the run setup.py
directly.
The image processing C extension requires the leptonica image processing
library (>=1.70, available from Debian Jessie):
```
$ apt-get install libleptonica-dev
```
Per default no dictionaries and OCR models (including data necessary to run
tests) are installed. To download the necessary files run:
```
$ python setup.py download
```
Tests
=====
```
$ python setup.py test
```
As mentioned above pip and the models have to be installed.
Running the tests requires a working tesseract with ancient greek language
files and an installed ocropus suite.
Running
=======
First edit (the installed) nidaba.yaml and celery.yaml to fit your needs. Have
a look at the [docs](https:///nidaba.rtfd.org/) if you haven't set up a
celery-based application before.
Then start up the celery daemon with something like:
```
$ celery -A nidaba worker
```
Next jobs can be added to the pipeline using the nidaba executable:
```
$ nidaba batch --binarize "sauvola:whsize=10;whsize=20;whsize=30;whsize=40,factor=0.6" --ocr tesseract:eng -- ./input.tiff
35be45e9-9d6d-47c7-8942-2717f00f84cb
```
Using the return code the current state of the job can be retrieved:
```
$ nidaba status 35be45e9-9d6d-47c7-8942-2717f00f84cb
PENDING
```
When the job has been processed the status command will return a list of paths
containing the final output:
```
$ nidaba status 35be45e9-9d6d-47c7-8942-2717f00f84cb
SUCCESS
/home/mittagessen/OCR/01c00777-ea8e-46e1-bc68-95023c7d29a1/input_rgb_to_gray_binarize_sauvola_10_0.3_ocr_tesseract_eng.tiff.hocr
/home/mittagessen/OCR/01c00777-ea8e-46e1-bc68-95023c7d29a1/input_rgb_to_gray_binarize_sauvola_20_0.3_ocr_tesseract_eng.tiff.hocr
/home/mittagessen/OCR/01c00777-ea8e-46e1-bc68-95023c7d29a1/input_rgb_to_gray_binarize_sauvola_30_0.3_ocr_tesseract_eng.tiff.hocr
/home/mittagessen/OCR/01c00777-ea8e-46e1-bc68-95023c7d29a1/input_rgb_to_gray_binarize_sauvola_40_0.3_ocr_tesseract_eng.tiff.hocr
```
Documentation
=============
Want to learn more? [Read the Docs](https:///ogl-nidaba.readthedocs.org/)
========
Nidaba is the central controller for the entire OGL OCR pipeline. It oversees and
automates the process of converting raw images into citable collections of
digitized texts. Images can be uploaded directly via Nidaba' RESTful web portal,
or can be selected from preexisting images located on Nidaba' image repository.
It offers the following functionality:
* Grayscale Conversion
* Binarization utilizing
[Sauvola](http://www.mediateam.oulu.fi/publications/pdf/24.p) adaptive
thresholding or leptonica's
[Otsu](http://www.leptonica.com/binarization.html) thresholding with
background normalization
* Deskewing
* Dewarping
* Integration of [tesseract](http://code.google.com/p/tesseract-ocr/) and ocropus OCR
engines
* Merging multiple hOCR documents using scoring
As it is designed to use a common storage medium on network attached storage
and the [celery](http://celeryproject.org) distributed task queue it scales
nicely to multi-machine clusters.
Build
=====
To build Nidaba run
```
$ pip install .
```
in the root directory. pip is required to extract the dependencies for
setuptools from the requirements file, so there is no reason the run setup.py
directly.
The image processing C extension requires the leptonica image processing
library (>=1.70, available from Debian Jessie):
```
$ apt-get install libleptonica-dev
```
Per default no dictionaries and OCR models (including data necessary to run
tests) are installed. To download the necessary files run:
```
$ python setup.py download
```
Tests
=====
```
$ python setup.py test
```
As mentioned above pip and the models have to be installed.
Running the tests requires a working tesseract with ancient greek language
files and an installed ocropus suite.
Running
=======
First edit (the installed) nidaba.yaml and celery.yaml to fit your needs. Have
a look at the [docs](https:///nidaba.rtfd.org/) if you haven't set up a
celery-based application before.
Then start up the celery daemon with something like:
```
$ celery -A nidaba worker
```
Next jobs can be added to the pipeline using the nidaba executable:
```
$ nidaba batch --binarize "sauvola:whsize=10;whsize=20;whsize=30;whsize=40,factor=0.6" --ocr tesseract:eng -- ./input.tiff
35be45e9-9d6d-47c7-8942-2717f00f84cb
```
Using the return code the current state of the job can be retrieved:
```
$ nidaba status 35be45e9-9d6d-47c7-8942-2717f00f84cb
PENDING
```
When the job has been processed the status command will return a list of paths
containing the final output:
```
$ nidaba status 35be45e9-9d6d-47c7-8942-2717f00f84cb
SUCCESS
/home/mittagessen/OCR/01c00777-ea8e-46e1-bc68-95023c7d29a1/input_rgb_to_gray_binarize_sauvola_10_0.3_ocr_tesseract_eng.tiff.hocr
/home/mittagessen/OCR/01c00777-ea8e-46e1-bc68-95023c7d29a1/input_rgb_to_gray_binarize_sauvola_20_0.3_ocr_tesseract_eng.tiff.hocr
/home/mittagessen/OCR/01c00777-ea8e-46e1-bc68-95023c7d29a1/input_rgb_to_gray_binarize_sauvola_30_0.3_ocr_tesseract_eng.tiff.hocr
/home/mittagessen/OCR/01c00777-ea8e-46e1-bc68-95023c7d29a1/input_rgb_to_gray_binarize_sauvola_40_0.3_ocr_tesseract_eng.tiff.hocr
```
Documentation
=============
Want to learn more? [Read the Docs](https:///ogl-nidaba.readthedocs.org/)
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
nidaba-0.1.0.post4.tar.gz
(61.4 kB
view details)
File details
Details for the file nidaba-0.1.0.post4.tar.gz.
File metadata
- Download URL: nidaba-0.1.0.post4.tar.gz
- Upload date:
- Size: 61.4 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 94e9eefc58e2627f255c5df515e35c3b4053d575618f8833119428d6edec5ffc |
|
| MD5 | 8d7dd1c03b3ab78aea49b6c29540957d |
|
| BLAKE2b-256 | 443d29fb961f285ad3be6b906df28ef9c2e455d92e3cb9d3354192361548737a |
