Toolset for computing directly on hierarchically nested, columnar data, such as Apache Arrow.
Project description
Large datasets can be more compact and faster to access when they are laid out in columns (see Apache Arrow). Even hierarchically nested data can be presented this way, though converting the data between the columnar form and the object form can degrade performance. Non-hierarchical data (rectangular tables) is often accessed without materializing rows (see Apache Drill), but this is more complex for data containing arbitrary-length lists of objects.
OAMap is a suite of tools for performing calculations in this way. The name stands for Object-Array-Map, in analogy with Object-Relational-Mapping (ORM) in relational databases. Pure Python calculations are considerably faster and more memory efficient when datasets are expressed in OAMaps, but the real power comes from compiling columized code. This toolset includes extensions to Numba that will compile your object-oriented code into native array manipulations. Generally, you’d use uncompiled Python for low latency exploration of the data and Numba-compiled functions for high throughput.
OAMap only strictly depends on Numpy, but Numba will accelerate it and pyarrow, h5py, etc. provide hooks for converting data among various formats.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
File details
Details for the file oamap-0.1.0.tar.gz.
File metadata
- Download URL: oamap-0.1.0.tar.gz
- Upload date:
- Size: 31.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | b9da8e4daa6fc2b558402bb2b9c71ec5edb7b00cc86c12c56626527836246cd8 |
|
| MD5 | 66c561c5a405a26554cca28bd8d62c90 |
|
| BLAKE2b-256 | 0bf3757cc9fb315fe60bfa99fe09f72ddaa1b3992599191cf66fadb930dcd5a4 |
