Skip to main content

Out-of-core NumPy arrays

Project description

Wendelin.core allows you to work with arrays bigger than RAM and local disk. Bigarrays are persisted to storage, and can be changed in transactional manner.

In other words bigarrays are something like numpy.memmap for numpy.ndarray and OS files, but support transactions and files bigger than disk. The whole bigarray cannot generally be used as a drop-in replacement for numpy arrays, but bigarray slices are real ndarrays and can be used everywhere ndarray can be used, including in C/Cython/Fortran code. Slice size is limited by virtual address-space size, which is ~ max 127TB on Linux/amd64.

The main class to work with is ZBigArray and is used like ndarray from NumPy:

  1. create array:

    from wendelin.bigarray.array_zodb import ZBigArray
    import transaction
    
    # root is connected to opened database
    root['A'] = A = ZBigArray(shape=..., dtype=...)
    transaction.commit()
  2. view array as a real ndarray:

    a = A[:]        # view which covers all array, if it fits into address-space
    b = A[10:100]

    data for views will be loaded lazily on memory access.

  3. work with views, including using C/Cython/Fortran functions from NumPy and other libraries to read/modify data:

    a[2] = 1
    a[10:20] = numpy.arange(10)
    numpy.mean(a)
    the amount of modifications in one transaction should be less than available RAM.
    the amount of data read is limited only by virtual address-space size.
  4. data can be appended to array in O(δ) time:

    values                  # ndarray to append of shape  (δ,)
    A.append(values)

    and array itself can be resized in O(1) time:

    A.resize(newshape)
  5. changes to array data can be either discarded or saved back to DB:

    transaction.abort()     # discard all made changes
    transaction.commit()    # atomically save all changes

When using NEO or ZEO as a database, bigarrays can be simultaneously used by several nodes in a cluster.

Please see demo/demo_zbigarray.py for a complete example.


Wendelin.core change history

0.6 (2016-06-13)

  • Add support for FORTRAN ordering (commit 1, 2)

  • Avoid deadlocks via doing loadblk() calls with virtmem lock released (commit 1, 2)

  • Various bugfixes

0.5 (2015-10-02)

  • Introduce another storage format, which is optimized for small changes, and make it the default. (commit 1, 2)

  • Various bugfixes and documentation improvements

0.4 (2015-08-19)

  • Add support for O(δ) in-place BigArray.append() (commit)

  • Implement proper multithreading support (commit)

  • Implement proper RAM pages invalidation when backing ZODB objects are changed from outside (commit 1, 2)

  • Fix all kind of failures that could happen when ZODB connection changes worker thread in-between handling requests (commit 1, 2)

  • Tox tests now cover usage with FileStorage, ZEO and NEO ZODB storages (commit 1, 2)

  • Various bugfixes

0.3 (2015-06-12)

  • Add support for automatic BigArray -> ndarray conversion, so that e.g. the following:

    A = BigArray(...)
    numpy.mean(A)       # passing BigArray to plain NumPy function

    either succeeds, or raises MemoryError if not enough address space is available to cover whole A. (current limitation is ~ 127TB on linux/amd64)

    (commit)

  • Various bugfixes (build-fixes, crashes, overflows, etc)

0.2 (2015-05-25)

  • Add support for O(1) in-place BigArray.resize() (commit)

  • Various build bugfixes (older systems, non-std python, etc)

0.1 (2015-04-03)

  • Initial release

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

wendelin.core-0.6.tar.gz (2.6 MB view details)

Uploaded Source

File details

Details for the file wendelin.core-0.6.tar.gz.

File metadata

  • Download URL: wendelin.core-0.6.tar.gz
  • Upload date:
  • Size: 2.6 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for wendelin.core-0.6.tar.gz
Algorithm Hash digest
SHA256 781c471d369ee25189a8f02206976e66e44f1d61cfb5e1f8a996bfa33249505f
MD5 6c37e3649aecb1fa585bf9f0250602d1
BLAKE2b-256 cdd34610095c248ab9183861b3ec231621572b8cb38d9ee74118e6dc892575b2

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page