Gutenberg 0.8.0

Python 3

This package depends on BSD-DB. The bsddb module was removed from the Python standard library since version 2.7. This means that if you wish to use gutenberg on Python 3, you will need to manually install BSD-DB.

sudo apt-get install libdb++-dev
export BERKELEYDB_DIR=/usr
pip install .

brew install berkeley-db4
pip install .

pip install bsddb3‑6.2.1‑cp35‑cp35m‑win_amd64.whl
pip install .

YES_I_HAVE_THE_RIGHT_TO_USE_THIS_BERKELEY_DB_VERSION=1

Apache Jena Fuseki

As an alternative to the BSD-DB backend, this package can also use Apache Jena Fuseki for the metadata store. The Apache Jena Fuseki backend is activated by setting the GUTENBERG_FUSEKI_URL environment variable to the HTTP endpoint at which Fuseki is listening. If the Fuseki server has HTTP basic authentication enabled, the username and password can be provided via the GUTENBERG_FUSEKI_USER and GUTENBERG_FUSEKI_PASSWORD environment variables.

For local development, the Fuseki server can be run via Docker:

docker run \
    --detach \
    --publish 3030:3030 \
    --env ADMIN_PASSWORD=some-password \
    --volume /some/mount/location:/fuseki \
    stain/jena-fuseki:3.6.0 \
    /jena-fuseki/fuseki-server --loc=/fuseki --update /ds

export GUTENBERG_FUSEKI_URL=http://localhost:3030/ds
export GUTENBERG_FUSEKI_USER=admin
export GUTENBERG_FUSEKI_PASSWORD=some-password

Downloading a text

from gutenberg.acquire import load_etext
from gutenberg.cleanup import strip_headers

text = strip_headers(load_etext(2701)).strip()
print(text)  # prints 'MOBY DICK; OR THE WHALE\n\nBy Herman Melville ...'

python -m gutenberg.acquire.text 2701 moby-raw.txt
python -m gutenberg.cleanup.strip_headers moby-raw.txt moby-clean.txt

Looking up meta-data

A bunch of meta-data about ebooks can be queried:

from gutenberg.query import get_etexts
from gutenberg.query import get_metadata

print(get_metadata('title', 2701))  # prints frozenset([u'Moby Dick; Or, The Whale'])
print(get_metadata('author', 2701)) # prints frozenset([u'Melville, Hermann'])

print(get_etexts('title', 'Moby Dick; Or, The Whale'))  # prints frozenset([2701, ...])
print(get_etexts('author', 'Melville, Hermann'))        # prints frozenset([2701, ...])

You can get a full list of the meta-data that can be queried by calling:

from gutenberg.query import list_supported_metadatas

print(list_supported_metadatas()) # prints (u'author', u'formaturi', u'language', ...)

Before you use one of the gutenberg.query functions you must populate the local metadata cache. This one-off process will take quite a while to complete (18 hours on my machine) but once it is done, any subsequent calls to get_etexts or get_metadata will be very fast. If you fail to populate the cache, the calls will raise an exception.

To populate the cache:

from gutenberg.acquire import get_metadata_cache
cache = get_metadata_cache()
cache.populate()

If you need more fine-grained control over the cache (e.g. where it’s stored or which backend is used), you can use the set_metadata_cache function to switch out the backend of the cache before you populate it. For example, to use the Sqlite cache backend instead of the default Sleepycat backend and store the cache at a custom location, you’d do the following:

from gutenberg.acquire import set_metadata_cache
from gutenberg.acquire.metadata import SqliteMetadataCache

cache = SqliteMetadataCache('/my/custom/location/cache.sqlite')
cache.populate()
set_metadata_cache(cache)