Flat-file datastore for timeseries data
Project description
PyStore - Datastore for timeseries data
PyStore is a simple (yet powerful) datastore for timeseries data. It’s built on top of Pandas, Numpy, Dask, and Parquet (via Fastparquet), to provide an easy to use datastore for Python developers that can easily query millions of rows per second per client.
PyStore is hugely inspired by Man AHL’s Arctic. I highly reommend you check it out.
Quickstart
Install PyStore
Install using pip:
$ pip install PyStoreOr upgrade using:
$ pip install PyStore --upgrade --no-cache-dirUsing PyStore
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import pystore
import quandl
# Connect to local datastore
store = pystore.Store('mydatastore')
# default path is `~/.pystore`, otherwise:
# store = pystore.Store('mydatastore', path='/usr/share/pystore')
# List existing collections
store.list_collections()
# Create a collection
store.create_collection('NASDAQ')
# Access the collection
collection = store.collection('NASDAQ')
# List items in collection
collection.list_items()
# Load some data from Quandl
aapl = quandl.get("WIKI/AAPL", authtoken="your token here")
# Store the first 100 rows of the data in the collection under "AAPL"
collection.write('AAPL', aapl[:100], metadata={'source': 'Quandl'})
# Reading the item's data
item = collection.item('AAPL')
data = item.data # <-- Dask dataframe (see dask.pydata.org)
metadata = item.metadata
df = item.to_pandas()
# Append the rest of the rows to the "AAPL" item
collection.append('AAPL', aapl[100:])
# Reading the item's data
item = collection.item('AAPL')
data = item.data
metadata = item.metadata
df = item.to_pandas()Concepts
PyStore provides namespaced collections of data. These collections allow bucketing data by source, user or some other metric (for example frequency: End-Of-Day; Minute Bars; etc.). Each collection (or namespace) maps to a directory containing partitioned parquet files for each item (e.g. symbol).
A good practice it to create collections that may look something like this:
collection.EOD
collection.ONEMINUTE
Known Limitation
PyStore currently only offers support for local filesystem. I plan on adding support for Amazon S3 (via s3fs), Google Cloud Storage (via gcsfs) and Hadoop Distributed File System (via hdfs3) in the future.
Requirements
PyStore currently works with:
Python 3.5 or higher
Pandas
Numpy
Dask
Fastparquet
Snappy (Google’s compression/decompression library)
Tested to work on:
Linux
Unix
macOS
Snappy Compression
PyStore uses Snappy, an extremely fast and efficient compression/decompression library from Google. You can install Snappy on *nix-like systems using your system’s package manager. For example, on Ubuntu, use sudo apt-get install libsnappy-dev, on macOS, use brew install snappy, etc.
Acknowledgements
PyStore is hugely inspired by Man AHL’s Arctic which uses MongoDB for storage and allow for versioning and other features. I highly reommend you check it out.
Contributions welcome!
License
PyStore is licensed under the GNU Lesser General Public License v2.1. A copy of which is included in LICENSE.txt.
I’m very interested in your experience with pystore. Please drop me an note with any feedback you have.
- Ran Aroussi
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
File details
Details for the file PyStore-0.0.5.tar.gz.
File metadata
- Download URL: PyStore-0.0.5.tar.gz
- Upload date:
- Size: 14.6 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 3a2cf55dff39d4b745dc7b8a9cd4d8f4d2f5b211b74cbf2f5905a76b0e9d67f0 |
|
| MD5 | 22daab7d821dcb1d438dc2e7dd877341 |
|
| BLAKE2b-256 | 88312be68367d80d09b7e65822fe2ab7b4a35308fbce0097bd1f2fd6024fcf87 |
