Fast and easy pandas and numpy data store
Project description
Arctic is a high performance datastore for numeric data. It supports Pandas, numpy arrays and pickled objects out-of-the-box, with pluggable support for other data types and optional versioning.
Arctic can query millions of rows per second per client, achieves ~10x compression on network bandwidth, ~10x compression on disk, and scales to hundreds of millions of rows per second per MongoDB instance.
Arctic has been under active development at Man AHL since 2012.
Quickstart
Install Arctic
pip install git+https://github.com/manahl/arctic.git
Run a MongoDB
mongod --dbpath <path/to/db_directory>
Using VersionStore
from arctic import Arctic
# Connect to Local MONGODB
store = Arctic('localhost')
# Create the library - defaults to VersionStore
store.initialize_library('NASDAQ')
# Access the library
library = store['NASDAQ']
# Load some data - maybe from Quandl
aapl = Quandl.get("NASDAQ/AAPL", authtoken="your token here")
# Store the data in the library
library.write('AAPL', aapl, metadata={'source': 'Quandl'})
# Reading the data
item = library.read('AAPL')
aapl = item.data
metadata = item.metadata
VersionStore supports much more: See the HowTo!
Adding your own storage engine
Plugging a custom class in as a library type is straightforward. This example shows how.
Concepts
Libraries
Arctic provides namespaced libraries of data. These libraries allow bucketing data by source, user or some other metric (for example frequency: End-Of-Day; Minute Bars; etc.).
Arctic supports multiple data libraries per user. A user (or namespace) maps to a MongoDB database (the granularity of mongo authentication). The library itself is composed of a number of collections within the database. Libraries look like:
user.EOD
user.ONEMINUTE
A library is mapped to a Python class. All library databases in MongoDB are prefixed with ‘arctic_’
Storage Engines
Arctic includes two storage engines:
VersionStore: a key-value versioned TimeSeries store. It supports:
Pandas data types (other Python types pickled)
Multiple versions of each data item. Can easily read previous versions.
Create point-in-time snapshots across symbols in a library
Soft quota support
Hooks for persisting other data types
Audited writes: API for saving metadata and data before and after a write.
a wide range of TimeSeries data frequencies: End-Of-Day to Minute bars
TickStore: Column oriented tick database. Supports dynamic fields, chunks aren’t versioned. Designed for large continuously ticking data.
Arctic storage implementations are pluggable. VersionStore is the default.
Requirements
Arctic currently works with:
Python 2.7
pymongo >= 3.0
Pandas
MongoDB >= 2.4.x
Acknowledgements
Arctic has been under active development at Man AHL since 2012.
It wouldn’t be possible without the work of the AHL Data Engineering Team including:
Tom Taylor
Tope Olukemi
Drake Siard
… and many others …
Contributions welcome!
License
Arctic is licensed under the GNU LGPL v2.1. A copy of which is included in LICENSE
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file arctic-1.1.0.tar.gz.
File metadata
- Download URL: arctic-1.1.0.tar.gz
- Upload date:
- Size: 116.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | a9913d8835c3997db647b5a6a3f617702a5f346437ec153136bbdf95d55af010 |
|
| MD5 | d33809f3ca8d048d9c940e3bd570ad36 |
|
| BLAKE2b-256 | 16f77cc5e5b39cf062da94434239cc2947031025c731d7435f966b6e0ae9f894 |
File details
Details for the file arctic-1.1.0-py2.7-linux-x86_64.egg.
File metadata
- Download URL: arctic-1.1.0-py2.7-linux-x86_64.egg
- Upload date:
- Size: 346.1 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | c2273baaadda78e72ba58e8c5c1972c035cf574ffdc096a471d068bdeb1dc50c |
|
| MD5 | 816f7c19b67953d77cccac15b407af3f |
|
| BLAKE2b-256 | 2e407961c5cfb447b3bf03ce5900a8239c88809cb3d3d209433a04e2f49f0a01 |
