Skip to main content

Lightweight pipelining: using Python functions as pipeline jobs.

Project description

Joblib is a set of tools to provide lightweight pipelining in Python. In particular:

  1. transparent disk-caching of functions and lazy re-evaluation (memoize pattern)

  2. easy simple parallel computing

Joblib is optimized to be fast and robust on large data in particular and has specific optimizations for numpy arrays. It is BSD-licensed.

Documentation:

https://joblib.readthedocs.io

Download:

https://pypi-hypernode.com/pypi/joblib#downloads

Source code:

https://github.com/joblib/joblib

Report issues:

https://github.com/joblib/joblib/issues

Vision

The vision is to provide tools to easily achieve better performance and reproducibility when working with long running jobs.

  • Avoid computing the same thing twice: code is often rerun again and again, for instance when prototyping computational-heavy jobs (as in scientific development), but hand-crafted solutions to alleviate this issue are error-prone and often lead to unreproducible results.

  • Persist to disk transparently: efficiently persisting arbitrary objects containing large data is hard. Using joblib’s caching mechanism avoids hand-written persistence and implicitly links the file on disk to the execution context of the original Python object. As a result, joblib’s persistence is good for resuming an application status or computational job, eg after a crash.

Joblib addresses these problems while leaving your code and your flow control as unmodified as possible (no framework, no new paradigms).

Main features

  1. Transparent and fast disk-caching of output value: a memoize or make-like functionality for Python functions that works well for arbitrary Python objects, including very large numpy arrays. Separate persistence and flow-execution logic from domain logic or algorithmic code by writing the operations as a set of steps with well-defined inputs and outputs: Python functions. Joblib can save their computation to disk and rerun it only if necessary:

    >>> from joblib import Memory
    >>> cachedir = 'your_cache_dir_goes_here'
    >>> mem = Memory(cachedir)
    >>> import numpy as np
    >>> a = np.vander(np.arange(3)).astype(np.float)
    >>> square = mem.cache(np.square)
    >>> b = square(a)                                   # doctest: +ELLIPSIS
    ________________________________________________________________________________
    [Memory] Calling square...
    square(array([[0., 0., 1.],
           [1., 1., 1.],
           [4., 2., 1.]]))
    ___________________________________________________________square - 0...s, 0.0min
    
    >>> c = square(a)
    >>> # The above call did not trigger an evaluation
  2. Embarrassingly parallel helper: to make it easy to write readable parallel code and debug it quickly:

    >>> from joblib import Parallel, delayed
    >>> from math import sqrt
    >>> Parallel(n_jobs=1)(delayed(sqrt)(i**2) for i in range(10))
    [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
  3. Fast compressed Persistence: a replacement for pickle to work efficiently on Python objects containing large data ( joblib.dump & joblib.load ).

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

joblib-0.15.1.tar.gz (347.5 kB view details)

Uploaded Source

Built Distribution

joblib-0.15.1-py3-none-any.whl (298.9 kB view details)

Uploaded Python 3

File details

Details for the file joblib-0.15.1.tar.gz.

File metadata

  • Download URL: joblib-0.15.1.tar.gz
  • Upload date:
  • Size: 347.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/46.1.3.post20200330 requests-toolbelt/0.9.1 tqdm/4.42.1 CPython/3.8.2

File hashes

Hashes for joblib-0.15.1.tar.gz
Algorithm Hash digest
SHA256 61e49189c84b3c5d99a969d314853f4d1d263316cc694bec17548ebaa9c47b6e
MD5 8760242e4719ca061aa7d5519a051e4b
BLAKE2b-256 57d245a038246a0596fb73af64c07e95578764d0fd115ce67f6b41eb457eed39

See more details on using hashes here.

Provenance

File details

Details for the file joblib-0.15.1-py3-none-any.whl.

File metadata

  • Download URL: joblib-0.15.1-py3-none-any.whl
  • Upload date:
  • Size: 298.9 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/46.1.3.post20200330 requests-toolbelt/0.9.1 tqdm/4.42.1 CPython/3.8.2

File hashes

Hashes for joblib-0.15.1-py3-none-any.whl
Algorithm Hash digest
SHA256 6825784ffda353cc8a1be573118085789e5b5d29401856b35b756645ab5aecb5
MD5 51200e114f61d3386c472594b08809f4
BLAKE2b-256 b8a6d1a816b89aa1e9e96bcb298eb1ee1854f21662ebc6d55ffa3d7b3b50122b

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page