Pythonic interface to the TileDB array storage manager
Project description
Array data management made fast and easy.
TileDB is an efficient multi-dimensional array management system which introduces a novel on-disk format that can effectively store dense and sparse array data with support for fast updates and reads. It also features excellent compression and an efficient parallel I/O system with high scalability.
TileDB-Py is the official Python interface to TileDB.
Quickstart
First, install TileDB-Py with pip:
$ pip install tiledb
This may take a while, as the pip package will automatically download and build the native TileDB library in addition to the Python bindings.
Next, save the quickstart program into a file and run it:
$ wget https://raw.githubusercontent.com/TileDB-Inc/TileDB-Py/dev/examples/quickstart_dense.py $ python quickstart_dense.py [[2 3 4] [6 7 8]]
The dense quickstart program simply creates a dense array on disk, writes some simple data to it, and reads a slice of the data back, printing the slice to the console.
Documentation
The full TileDB documentation can be found at https://docs.tiledb.io and includes many tutorials and examples to get you started.
The latest Python API reference can be found at https://docs.tiledb.io/projects/tiledb-py/en/latest/python-api.html.
Installation
Pip
A PyPI package is available which can be installed with Pip. This package will download and install the native TileDB library inside the site package if TileDB is not already installed on your system.
$ pip install tiledb
Note: if the Numpy and Cython dependencies are not installed, pip will try to build them from source. This can take a long time and make the install appear to “hang.” Pass the -v flag to pip to monitor the build process.
If you wish to use a custom version of the TileDB library and the install location is not in the compiler search path, create a requirements.txt file that specifies the tiledb install path manually.
$ cat > tiledb_requirements.txt <<EOF tiledb==<version> --install-option="--tiledb=<path/to/tiledb/install>" EOF $ pip install -r tiledb_requirements.txt
Do not forget to put the built .so / .dylib / .dll on the dynamic linker path, otherwise TileDB-Py will fail to load the shared library upon import.
Conda Package
A pre-built Conda package is available that will install TileDB as well.
$ conda install -c conda-forge tiledb-py
Note: Currently the pre-built TileDB conda package does not include the HDFS storage backend.
Installing From Source
TileDB-Py Build Dependencies
Numpy
Cython
C++11 compiler
CMake
TileDB-Py Runtime Dependencies
Numpy
Linux / OSX
Simply execute the following commands:
$ git clone https://github.com/TileDB-Inc/TileDB-Py.git $ cd TileDB-Py $ pip install -r requirements_dev.txt $ python setup.py build_ext --inplace $ python setup.py install
If you wish to use a custom version of the TileDB library and it is installed in a non-standard location, pass the path to setup.py with the --tiledb= flag. If you want to pass extra compiler/linker flags during the C++ extension compilation step use --cxxflags= or --lflags=.
$ python setup.py build_ext --inplace --tiledb=/home/tiledb/dist
If TileDB is installed in a non-standard location, you also need to make the dynamic linker aware of libtiledb’s location. Otherwise when importing the tiledb module you will get an error that the built extension module cannot find libtiledb’s symbols:
$ env LD_LIBRARY_PATH="/home/tiledb/dist/lib:$LD_LIBRARY_PATH" python -m unittest -v
For macOS the linker environment variable is DYLD_LIBRARY_PATH
Installing on Windows
If you are building the extension on Windows, first install a Python distribution such as Miniconda. You can then either build TileDB from source, or download the pre-built binaries.
Once you’ve installed Miniconda and TileDB, open the Miniconda command prompt and execute:
> cd TileDB-Py > conda install conda-build > conda install virtualenv > virtualenv venv > venv\Scripts\activate > pip install -r requirements_dev.txt > python setup.py build_ext --inplace --tiledb=C:\path\to\TileDB\ > set PATH=%PATH%;C:\path\to\TileDB\bin > python -m unittest -v
Note that if you built TileDB from source, then replace C:\path\to\TileDB with C:\path\to\TileDB\dist.
Developing and testing TileDB-Py
TileDB-Py includes a handy Conda environment definition file for setting up a test environment:
$ conda env create -f environment.yml
This will create a tiledbpy conda environment with all the development library dependencies.
The easiest way to test / develop TileDB-Py across Python versions (2.7, 3.5, and 3.6), is using tox. TileDB includes a tox.ini file, simply run tox in the toplevel source directory to run the test suite against multiple installed Python versions:
$ tox
You can specify a particular Python version using the -e flag:
$ tox -e py27
If TileDB is not installed in a global system location, you must specify the install path to tox:
$ env TILEDB_PATH=/path/to/tiledb LD_LIBRARY_PATH=/path/to/tiledb/libdir:${LD_LIBRARY_PATH} tox
You can also run the unittests from the source folder without having the package installed. First build the package in place from the source directory:
$ python setup.py build_ext --inplace
Tests can now be run using Python’s unittest framework:
$ python -m unittest -v
Doctests can be run using the doctest module:
$ python -m doctest -o NORMALIZE_WHITESPACE -f tiledb/libtiledb.pyx
You can also install a symlink named site-packages/tiledb.egg-link to the development folder of TileDB-Py with:
$ pip install --editable .
This enables local changes to the current development repo to be reflected globally.
Building TileDB-Py against libtiledb from conda
TileDB core (libtiledb) is distributed as a separate conda package, [tiledb](https://anaconda.org/conda-forge/tiledb), which may be used for local builds of TileDB-Py:
activate the desired conda environment
run conda install tiledb
run python setup.py build_ext –inplace –tiledb=$CONDA_PREFIX
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
File details
Details for the file tiledb-0.5.0.tar.gz.
File metadata
- Download URL: tiledb-0.5.0.tar.gz
- Upload date:
- Size: 95.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.6.0.post20191029 requests-toolbelt/0.9.1 tqdm/4.32.2 CPython/3.7.3
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | d76226bef676275b310ba48192f82f24b85160456ddf808d8fd4db5c11414931 |
|
| MD5 | 800e31900135a384892af486dd61b840 |
|
| BLAKE2b-256 | 7a9b40907ba51f1a94fad231c1091bdc6c94f99869c04bb1422858e36cc19e20 |
Provenance
File details
Details for the file tiledb-0.5.0-cp37-cp37m-manylinux1_x86_64.whl.
File metadata
- Download URL: tiledb-0.5.0-cp37-cp37m-manylinux1_x86_64.whl
- Upload date:
- Size: 7.1 MB
- Tags: CPython 3.7m
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.6.0.post20191029 requests-toolbelt/0.9.1 tqdm/4.32.2 CPython/3.7.3
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 54b422a9845a22072fe8d5e57120e1648ee2b779cea43cf56eb2f39874b98765 |
|
| MD5 | 28d41cde5d00bfc4d3a2ec3d577a222e |
|
| BLAKE2b-256 | f1052833983b7f2a4f2c6deaf588be8cf6ee7625e8f149725b235039d98f3861 |
Provenance
File details
Details for the file tiledb-0.5.0-cp36-cp36m-manylinux1_x86_64.whl.
File metadata
- Download URL: tiledb-0.5.0-cp36-cp36m-manylinux1_x86_64.whl
- Upload date:
- Size: 7.1 MB
- Tags: CPython 3.6m
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.6.0.post20191029 requests-toolbelt/0.9.1 tqdm/4.32.2 CPython/3.7.3
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 05a8ff468a5b9ce9c8aa9e9f5f9a4c207e0a846535bd9514ee5788d8f9c56bde |
|
| MD5 | 279ac3daf1b5384d6a2e359cb393c7ca |
|
| BLAKE2b-256 | 9a4289fb77423b192b8f5784da6506e25ce3f3814801eb51514fa5557f9eb22a |
Provenance
File details
Details for the file tiledb-0.5.0-cp35-cp35m-manylinux1_x86_64.whl.
File metadata
- Download URL: tiledb-0.5.0-cp35-cp35m-manylinux1_x86_64.whl
- Upload date:
- Size: 7.0 MB
- Tags: CPython 3.5m
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.6.0.post20191029 requests-toolbelt/0.9.1 tqdm/4.32.2 CPython/3.7.3
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 8be690bd70107b8b513b2fbba5997700ee1957c43548373bbab8559197a24239 |
|
| MD5 | 46e48125b01a8f5032e8539076fe0624 |
|
| BLAKE2b-256 | e5fb661738abc3001ae8dbc069d1cb6c4d0f196dbe3e37316ca17bdc0dffd827 |
Provenance
File details
Details for the file tiledb-0.5.0-cp27-cp27mu-manylinux1_x86_64.whl.
File metadata
- Download URL: tiledb-0.5.0-cp27-cp27mu-manylinux1_x86_64.whl
- Upload date:
- Size: 6.8 MB
- Tags: CPython 2.7mu
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.6.0.post20191029 requests-toolbelt/0.9.1 tqdm/4.32.2 CPython/3.7.3
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | fb7336acad7a3098e4e02f2aea55169f342e3eb94973f9d74ee59806d36b42f2 |
|
| MD5 | a0cc6bf6e2def0ddb5dfa01bce1abc1a |
|
| BLAKE2b-256 | d101b0e93c27fe7733899a06b29df73a80dfb387a21e8c15b22e99286b3364c2 |
