Skip to main content

ArcticDB DataFrame Database

Project description



ArcticDB Website | ArcticDB Blog | Press Release | Press Release | Community


ArcticDB is a high performance, serverless DataFrame database built for the Python Data Science ecosystem. Launched in March 2023, it is the successor to Arctic.

ArcticDB offers an intuitive Python-centric API enabling you to read and write Pandas DataFrames to S3 or LMDB utilising a fast C++ data-processing and compression engine.

ArcticDB allows you to:

  • Pandas in, Pandas out: Read and write Pandas DataFrames, NumPy arrays and native types to S3 and LMDB without leaving Python.
  • Built for time-series data: Efficiently index and query time-series data across billions of rows
  • Time travel: Travel back in time to see previous versions of your data and create customizable snapshots of the database
  • Schemaless Database: Append, update and modify data without being constrained by the existing schema
  • Optimised for streaming data: Built in support for efficient sparse data storage
  • Powerful processing: Filter, aggregate and create new columns on-the-fly with a Pandas-like syntax
  • C++ efficiency: Accelerate analytics though concurrency in the C++ data-processing engine

ArcticDB handles data that is big in both row count and column count, so a 20-year history of more than 400,000 unique securities can be stored in a single symbol. Each symbol is maintained as a separate entity with no shared data which means ArcticDB can scale horizontally across symbols, maximising the performance potential of your compute, storage and network.

ArcticDB is designed from the outset to be resilient; there is no single point of failure, and persistent data structures in the storage mean that once a version of a symbol has been written, it can never be corrupted by subsequent updates. Pulling compressed data directly from storage to the client means that there is no server to overload, so your data is always available when you need it.

Quickstart

Prebuilt binary availability

PyPI (Python 3.6 - 3.11) conda-forge (Python 3.8 - 3.11)
Linux ✔️ ✔️
Windows ✔️
MacOS (Apple Silicon) ✔️

Storage compatibility

PyPI conda-forge
S3 ✔️ ✔️
LMDB ✔️ ✔️
Azure Blob Storage ✔️

Support for Azure Blob Storage in conda-forge is tracked in #519.

Installation

Install ArcticDB:

$ pip install arcticdb

or using conda-forge

$ conda install -c conda-forge arcticdb

Import ArcticDB:

>>> from arcticdb import Arctic

Create an instance on your S3 storage (with or without explicit credentials):

# Leave AWS to derive credential information
>>> ac = Arctic('s3://MY_ENDPOINT:MY_BUCKET?aws_auth=true')

# Manually specify creds
>>> ac = Arctic('s3://MY_ENDPOINT:MY_BUCKET?region=YOUR_REGION&access=ABCD&secret=DCBA')

Or create an instance on your local disk:

>>> ac = Arctic("lmdb:///<path>")

Create your first library and list the libraries in the instance:

>>> ac.create_library('travel_data')
>>> ac.list_libraries()

Create a test dataframe:

>>> import numpy as np
>>> import pandas as pd
>>> NUM_COLUMNS=10
>>> NUM_ROWS=100_000
>>> df = pd.DataFrame(np.random.randint(0,100,size=(NUM_ROWS, NUM_COLUMNS)), columns=[f"COL_{i}" for i in range(NUM_COLUMNS)], index=pd.date_range('2000', periods=NUM_ROWS, freq='h'))

Get the library, write some data to it, and read it back:

>>> lib = ac['travel_data']
>>> lib.write("my_data", df)
>>> data = lib.read("my_data")

To find out more about working with data, visit our docs


Documentation

The source code for the ArcticDB docs are located in the docs folder, and are hosted at docs.arcticdb.io.

License

ArcticDB is released under a Business Source License 1.1 (BSL)

BSL features are free to use and the source code is available, but users may not use ArcticDB for production use or for a Database Service, without agreement with Man Group Operations Limited.

Use of ArcticDB in production or for a Database Service requires a paid for license from Man Group Operations Limited and is licensed under the ArcticDB Software License Agreement. For more information please contact arcticdb@man.com.

The BSL is not certified as an open-source license, but most of the Open Source Initiative (OSI) criteria are met. Please see version conversion dates in the below table:

ArcticDB Version License Converts to Apache 2.0
1.0 Business Source License 1.1 Mar 16, 2025
1.2 Business Source License 1.1 May 22, 2025
1.3 Business Source License 1.1 Jun 9, 2025
1.4 Business Source License 1.1 Jun 23, 2025

Code of Conduct

Code of Conduct

This project has adopted a Code of Conduct. If you have any concerns about the Code, or behaviour that you have experienced in the project, please contact us at arcticdb@man.com.

Contributing/Building From Source

We welcome your contributions to help us improve and extend this project!

Please refer to the Contributing page and feel free to open issues on GitHub.

We are also always looking for feedback from our dedicated community! If you have used ArcticDB please let us know, we would love to hear about your experience!

Our release process is documented here.

Community

We would love to hear how your ArcticDB journey evolves, email us at arcticdb@man.com or come chat to us on Twitter!

Interested in learning more about ArcticDB? Head over to our blog!

Do you have any questions or issues? Chat to us and other users through our dedicated Slack Workspace - sign up for Slack access on our website.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

arcticdb-1.5.0-cp311-cp311-win_amd64.whl (6.5 MB view details)

Uploaded CPython 3.11 Windows x86-64

arcticdb-1.5.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.9 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

arcticdb-1.5.0-cp310-cp310-win_amd64.whl (6.5 MB view details)

Uploaded CPython 3.10 Windows x86-64

arcticdb-1.5.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.9 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

arcticdb-1.5.0-cp39-cp39-win_amd64.whl (6.5 MB view details)

Uploaded CPython 3.9 Windows x86-64

arcticdb-1.5.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.9 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

arcticdb-1.5.0-cp38-cp38-win_amd64.whl (6.5 MB view details)

Uploaded CPython 3.8 Windows x86-64

arcticdb-1.5.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.9 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

arcticdb-1.5.0-cp37-cp37m-win_amd64.whl (6.5 MB view details)

Uploaded CPython 3.7m Windows x86-64

arcticdb-1.5.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (14.0 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64

arcticdb-1.5.0-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (14.0 MB view details)

Uploaded CPython 3.6m manylinux: glibc 2.17+ x86-64

File details

Details for the file arcticdb-1.5.0-cp311-cp311-win_amd64.whl.

File metadata

File hashes

Hashes for arcticdb-1.5.0-cp311-cp311-win_amd64.whl
Algorithm Hash digest
SHA256 6a3f7d16d9ebcd36e19eb00196190a18089714f8f638616414e9e890f73ea675
MD5 7da5e0c04d0153c904844300be0e7b72
BLAKE2b-256 c48fd9904e112d2d42d2e496c8d1fe73a0a1ac650aa8def0d4e3902a48daf717

See more details on using hashes here.

Provenance

File details

Details for the file arcticdb-1.5.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for arcticdb-1.5.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 f77f1b91cd1e91cea9473f4caf927e6b66670cd8beb5dfd82b1c2a5266ca37f6
MD5 8b3b1629f5c7e2af4483bba01d6147b3
BLAKE2b-256 f0051511030ddef6fc85da05f20114794a900c968fc94b389cd12c752d439804

See more details on using hashes here.

Provenance

File details

Details for the file arcticdb-1.5.0-cp310-cp310-win_amd64.whl.

File metadata

File hashes

Hashes for arcticdb-1.5.0-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 7cc9d68ea112227a40a7af6d9d87ec5f4b54597f2a3d627ac0213743c6a96710
MD5 12d7eabff4dfa6ba5c8707ff0c9bcb81
BLAKE2b-256 7dcfb28b36dc9c82237558fca8bb1b8dac0b59a2e99a8d15f653b48644359407

See more details on using hashes here.

Provenance

File details

Details for the file arcticdb-1.5.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for arcticdb-1.5.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 c63939fc1b1fc3eef70f46f4eea24f937173cb64562dd3bd6295db2dfd4fc9b1
MD5 25696a9641af64baef3f75fd1495b22e
BLAKE2b-256 d6eca3051700f1927fd62904141d62a36e13046fbc6d3ea8fce5971effff6ebf

See more details on using hashes here.

Provenance

File details

Details for the file arcticdb-1.5.0-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: arcticdb-1.5.0-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 6.5 MB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.10.6

File hashes

Hashes for arcticdb-1.5.0-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 63cceef6402c3f794bd89e4f2cc926a0761ef95c000f984ee3728153a3073837
MD5 1df68ea82f02bc3c12257f2d20f56e24
BLAKE2b-256 10d234a3c5bc8a032be4fbd405dbbaa8c0260f4c7c883676b4aee94f265def9b

See more details on using hashes here.

Provenance

File details

Details for the file arcticdb-1.5.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for arcticdb-1.5.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 75284e8ee6dd18823714a4dc3d5aa1f33243ee001de83f9335117795e7bd4797
MD5 7766317cb5fea749636035d5e0d20766
BLAKE2b-256 c3fa663f1dfcb8577ea9e83872b6ac061efd479aff9698dec25418e1ebea7e69

See more details on using hashes here.

Provenance

File details

Details for the file arcticdb-1.5.0-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: arcticdb-1.5.0-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 6.5 MB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.10.6

File hashes

Hashes for arcticdb-1.5.0-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 9150ce11e5a11dd0c3e2cdc038ae4cfca7545f7dfa43cb996600f086a2a9cc35
MD5 d46997e1a03e1a8ed144da704401402e
BLAKE2b-256 3283a2dc38c4d49d6b50dc3c83270565273f0251ab58c76c17f5b96fc9d45e01

See more details on using hashes here.

Provenance

File details

Details for the file arcticdb-1.5.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for arcticdb-1.5.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 eb9d7f3dd252302717192c4a713143eafd8c484687786cde841f980d17b38dd0
MD5 c77e2d647b075bdba944e326111416a0
BLAKE2b-256 f586dcf8dae9aaffc68b469c39eaba4fef3d3c7a212b1593d604f9d53aa8f93a

See more details on using hashes here.

Provenance

File details

Details for the file arcticdb-1.5.0-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: arcticdb-1.5.0-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 6.5 MB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.10.6

File hashes

Hashes for arcticdb-1.5.0-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 541325cc16aa352f8b680950dec4a448a677c519b55358f9be1f47926a695687
MD5 83bb3a51fa8d1c0812b6b4a339c77e50
BLAKE2b-256 9767b56041fd8b7b860a9d03b9a6b2adcb8cdf9cd256a3e9c07eede063b31914

See more details on using hashes here.

Provenance

File details

Details for the file arcticdb-1.5.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for arcticdb-1.5.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 8e632e5969f32ea05981062173b77716d9f808a3901eecce65cd3b10c7105130
MD5 294905d7707a59b066232f4f44d26ef0
BLAKE2b-256 74a1fd03628cbc9ca1b0ea46344408f5f05de6b063b4c320362acbd83114a33a

See more details on using hashes here.

Provenance

File details

Details for the file arcticdb-1.5.0-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for arcticdb-1.5.0-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 c32e5e0416b04f6ff48ef16b5636a9898fd1a35cff916bff06519d5a8940f5ed
MD5 378c251b8953d2923dbed2460958d9a8
BLAKE2b-256 ca0d3eb5e05f8001104519a5c225acbf5da91b46a71a7832f07d318314207e25

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page