Skip to main content

Simple File-based KV-Store

Project description

kvfile

Travis Coveralls

A simple Key-Value store that's file based - so can accommodate large data sets with a small memory footprint.

Internally will use the faster leveldb as a storage backend or sqlite3 as fallback if leveldb is not available.

The Basics

The API should feel familiar to anyone working with Python. It exposes get, keys and items for reading from the DB, and set for setting a value in the DB.

Initializing

import datetime
import decimal

from kvfile import KVFile

kv = KVFile()

Setting values

kv.set('s', 'value')
kv.set('i', 123)
kv.set('d', datetime.datetime.fromtimestamp(12325))
kv.set('n', decimal.Decimal('1234.56'))
kv.set('ss', set(range(10)))
kv.set('o', dict(d=decimal.Decimal('1234.58'), 
                 n=datetime.datetime.fromtimestamp(12325)))

Getting values

assert kv.get('s') == 'value'
assert kv.get('i') == 123
assert kv.get('d') == datetime.datetime.fromtimestamp(12325)
assert kv.get('n') == decimal.Decimal('1234.56')
assert kv.get('ss') == set(range(10))
assert kv.get('o') == dict(d=decimal.Decimal('1234.58'), 
                           n=datetime.datetime.fromtimestamp(12325))

Listing values

keys() and items() methods return a generator yielding the values for efficient stream processing.

The returned data is sorted ascending (by default) based on the keys

assert list(kv.keys()) == ['d', 'i', 'n', 'o', 's', 'ss']
assert list(kv.items()) == [
  ('d', datetime.datetime.fromtimestamp(12325)), 
  ('i', 123), 
  ('n', decimal.Decimal('1234.56')), 
  ('o', {'d': decimal.Decimal('1234.58'), 
         'n': datetime.datetime.fromtimestamp(12325)}), 
  ('s', 'value'), 
  ('ss', {0, 1, 2, 3, 4, 5, 6, 7, 8, 9})
]

Set the reverse argument to True for the keys() and items() methods to sort in descending order.

Bulk inserting data

The SQLite DB backend can be very slow when bulk inserting data. You can use the insert method to insert efficiently in bulk.

kv.insert(((str(i), ':{}'.format(i)) for i in range(50000)))

The batch size is 1000 by default, you should modify it depending on the size of your data and available memory.

kv.insert(((str(i), ':{}'.format(i)) for i in range(50000)), batch_size=40000)

If you are inserting data from a generator and need to use the inserted data, use insert_generator method:

for key, value in kv.insert_generator(((str(i), ':{}'.format(i)) for i in range(50)), batch_size=10):
    print(key, value)

Installing leveldb

On Debian based Linux:

$ apt-get install libleveldb-dev libleveldb1

On Alpine based Linux:

$ apk --repository http://dl-3.alpinelinux.org/alpine/edge/testing/ --update add leveldb leveldb-dev

On OS X:

$ brew install leveldb

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

kvfile-0.0.7.tar.gz (7.4 kB view details)

Uploaded Source

Built Distribution

kvfile-0.0.7-py2.py3-none-any.whl (6.3 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file kvfile-0.0.7.tar.gz.

File metadata

  • Download URL: kvfile-0.0.7.tar.gz
  • Upload date:
  • Size: 7.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.20.1 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.7.0

File hashes

Hashes for kvfile-0.0.7.tar.gz
Algorithm Hash digest
SHA256 cf23ebd5dfecb40aa66289f979497901bece2c006cfd859a6fc1510b9ac65d92
MD5 9c0341d07056e43f5bd6462529dca3e3
BLAKE2b-256 c41e7a313bdb88247109463018bfeecd8f69880fdbf38b4876177b198d4b255f

See more details on using hashes here.

Provenance

File details

Details for the file kvfile-0.0.7-py2.py3-none-any.whl.

File metadata

  • Download URL: kvfile-0.0.7-py2.py3-none-any.whl
  • Upload date:
  • Size: 6.3 kB
  • Tags: Python 2, Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.20.1 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.7.0

File hashes

Hashes for kvfile-0.0.7-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 05524d84e1d94c9455e44fd21a0c97da7b582b7bd3883b4df0ca12a9efcffc50
MD5 317e64761ba789b7e8e26a89f6d42876
BLAKE2b-256 1faebfb2ac5a622b0257b3e70caabbea42e7252ea825c1474145dbbe258799c3

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page