ArcticDB DataFrame Database
Project description
ArcticDB is a high performance, serverless DataFrame database built for the Python Data Science ecosystem. Launched in February 2023, it is the successor to Arctic. ArcticDB offers an intuitive Python-centric API, with a fast C++ data-processing and compression engine that is compatible with object stores such as AWS S3, making it suitable for public cloud and on-premise deployments.
ArcticDB allows you to:
- Use standard data types and integrate effortlessly with the Python data science ecosystem - Pandas in, Pandas out
- Efficiently index and query time-series data
- Store tiled dataframes, for fast selection of rows and columns
- Travel back in time to see previous versions of your data and create customizable snapshots of the database
- Append and update data without being constrained by the existing schema
- Handle sparse values and missing columns
- Filter, aggregate and create new columns on-the-fly with a Pandas-like syntax
- Accelerate analytics though concurrency in the C++ data-processing engine
ArcticDB handles data that is big in both row count and column count, so a 20-year history of more than 400,000 unique securities can be stored in a single symbol. Each symbol is maintained as a separate entity with no shared data which means ArcticDB can scale horizontally across symbols, maximising the peformance potential of your compute, storage and network.
ArcticDB is designed from the outset to be resilient; there is no single point of failure, and persistent data structures in the storage mean that once a version of a symbol has been written, it can never be corrupted by subsequent updates. Pulling compressed data directly from storage to the client means that there is no server to overload, so your data is always available when you need it.
Quickstart
Install ArcticDB:
$ pip install arcticdb
Import ArcticDB:
>>> from arcticdb import Arctic
Create an instance on your S3 storage (with or without explicit credentials):
>>> ac = Arctic('s3://MY_ENDPOINT:MY_BUCKET?aws_auth=true') # Leave AWS to derive credential information
>>> ac = Arctic('s3://MY_ENDPOINT:MY_BUCKET?region=YOUR_REGION&access=ABCD&secret=DCBA') # Manually specify creds
Or create an instance on your local disk:
>>> ac = Arctic("lmdb:///<path>)
Create your first library and list the libraries in the instance:
>>> ac.create_library('travel_data')
>>> ac.list_libraries()
Create a test dataframe:
>>> NUM_COLUMNS=10
>>> NUM_ROWS=100_000
>>> df = pd.DataFrame(np.random.randint(0,100,size=(NUM_ROWS, NUM_COLUMNS)), columns=[f"COL_{i}" for i in range(NUM_COLUMNS)], index=pd.date_range('2000', periods=NUM_ROWS, freq='h'))
Get the library, write some data to it, and read it back:
>>> lib = ac['travel_data']
>>> lib.write("my_data", df)
>>> data = lib.read("my_data")
To find out more about working with data, visit our docs
Build From Source
Instructions for building from source coming soon.
Documentation
The source code for the ArcticDB docs are located in the docs folder, and are hosted at docs.arcticdb.io.
License
ArcticDB is released under a Business Source License 1.1 (BSL)
BSL features are free to use and the source code is available, but users may not use ArcticDB for production use or for a Database Service, without agreement with Man Group Operations Limited.
Use of ArcticDB in production or for a Database Service requires a paid for license from Man Group Operations Limited and is licensed under the ArcticDB Software License Agreement. For more information please contact arcticdb@man.com.
The BSL is not certified as an open-source license, but most of the Open Source Initiative (OSI) criteria are met.
For each BSL release all associated alpha, beta, major, and minor (point) releases will become Apache Lisensed, version 2.0 on the same day two years after the major release date. For the license conversion dates, see the table below.
| ArcticDB Version | License | Converts to Apache 2.0 |
|---|---|---|
| 1.0 | Business Source License 1.1 | Mar 16, 2025 |
Code of Conduct
This project has adopted a Code of Conduct. If you have any concerns about the Code, or behaviour that you have experienced in the project, please contact us at arcticdb@man.com.
Contributing
We welcome your contributions to help us improve and extend this project!
Below you will find some basic steps required to be able to contribute to the project. If you have any questions about this process or any other aspect of contributing to our project, feel free to send an email to arcticdb@man.com and we'll get your questions answered as quickly as we can.
We are also always looking for feedback from our dedicated community! If you have used ArcticDB please let us know, we would love to hear about your experience!
Contribution Licensing
Since this project is distributed under the terms of the BSL license, contributions that you make are licensed under the same terms. For us to be able to accept your contributions, we will need explicit confirmation from you that you are able and willing to provide them under these terms, and the mechanism we use to do this is the ArcticDB Individual Contributor License Agreement.
Individuals - To participate under these terms, please include the following line as the last line of the commit message for each commit in your contribution. You must use your real name (no pseudonyms, and no anonymous contributions).
Signed-Off By: Random J. Developer random@developer.example.org. By including this sign-off line I agree to the terms of the Contributor License Agreement.
Corporations - For corporations who wish to make contributions to ArcticDB, please contact arcticdb@man.com and we will arrange for the CLA to be sent to the signing authority within your corporation.
Community
Do you have any questions or issues? Chat to us and other users through our dedicated Slack Workspace - sign up for Slack access on our website.
Alternatively email us at arcticdb@man.com or come chat to us on Twitter!
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
File details
Details for the file arcticdb-1.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.
File metadata
- Download URL: arcticdb-1.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Upload date:
- Size: 13.0 MB
- Tags: CPython 3.10, manylinux: glibc 2.17+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.25.1 setuptools/58.0.4 requests-toolbelt/0.8.0 tqdm/4.35.0 CPython/3.6.15
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 40d40bca8dc99d6dc4d5d3a2a3cdfa0eef4a746521aff3faeac421c731ebd371 |
|
| MD5 | 42065b275d444a972a9dd71da9944704 |
|
| BLAKE2b-256 | 1ed9779081c15d4c5f3db011898d0d80dd1d7aecec8f2409ab4be96c1d618988 |
Provenance
File details
Details for the file arcticdb-1.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.
File metadata
- Download URL: arcticdb-1.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Upload date:
- Size: 13.0 MB
- Tags: CPython 3.9, manylinux: glibc 2.17+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.25.1 setuptools/58.0.4 requests-toolbelt/0.8.0 tqdm/4.35.0 CPython/3.6.15
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 402d705671fa8dec152db488d5fbb7cf2f053701eca67ef80e1dcc65178b2965 |
|
| MD5 | dc7c5378eb94c91f28072e8aaade6ec5 |
|
| BLAKE2b-256 | a4ee9c1392e04cd81442e0e1ec55d5e00f95495154ad4ba027c5ccf1bd0ab44f |
Provenance
File details
Details for the file arcticdb-1.0.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.
File metadata
- Download URL: arcticdb-1.0.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Upload date:
- Size: 13.0 MB
- Tags: CPython 3.8, manylinux: glibc 2.17+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.25.1 setuptools/58.0.4 requests-toolbelt/0.8.0 tqdm/4.35.0 CPython/3.6.15
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 7194b3daf33fc3bb36d5bbc1f043a6e3b96c133425944b665472f4ed8b231511 |
|
| MD5 | 9192a93f8d649ba0469339499418a9a1 |
|
| BLAKE2b-256 | 9480710603c699123cf3ee8adff1816415340ea6b6981ad1c3039986842747d1 |
Provenance
File details
Details for the file arcticdb-1.0.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.
File metadata
- Download URL: arcticdb-1.0.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Upload date:
- Size: 13.0 MB
- Tags: CPython 3.7m, manylinux: glibc 2.17+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.25.1 setuptools/58.0.4 requests-toolbelt/0.8.0 tqdm/4.35.0 CPython/3.6.15
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | f543fae4380733400a40e212b44f14a97d157cb30ae54b9c117ee0d91e866cc6 |
|
| MD5 | 28b908bb94fbe0570fa2e48371fd6e87 |
|
| BLAKE2b-256 | 429696346be76b62198342d6370e29e9ce36194d423fe6513e636b27e2599495 |
Provenance
File details
Details for the file arcticdb-1.0.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.
File metadata
- Download URL: arcticdb-1.0.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Upload date:
- Size: 13.0 MB
- Tags: CPython 3.6m, manylinux: glibc 2.17+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.25.1 setuptools/58.0.4 requests-toolbelt/0.8.0 tqdm/4.35.0 CPython/3.6.15
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 01bb0afd3a0f29e8f086666a29d8ef7663f1dbabda81ddb824d33016e29e75a5 |
|
| MD5 | 2b56546daf72baedcc89de03f1ae7661 |
|
| BLAKE2b-256 | 2dc340cca4484665a92d1b1b80e31d5ce85bff99388f9ed38539f5755699d32c |
