Skip to main content

Simple File-based KV-Store

Project description

kvfile

Travis Coveralls

A simple Key-Value store that's file based - so can accommodate large data sets with a small memory footprint.

Internally will use the faster leveldb as a storage backend or sqlite3 as fallback if leveldb is not available.

The Basics

The API should feel familiar to anyone working with Python. It exposes get, keys and items for reading from the DB, and set for setting a value in the DB.

Initializing

import datetime
import decimal

from kvfile import KVFile

kv = KVFile()

Setting values

kv.set('s', 'value')
kv.set('i', 123)
kv.set('d', datetime.datetime.fromtimestamp(12325))
kv.set('n', decimal.Decimal('1234.56'))
kv.set('ss', set(range(10)))
kv.set('o', dict(d=decimal.Decimal('1234.58'), 
                 n=datetime.datetime.fromtimestamp(12325)))

Getting values

assert kv.get('s') == 'value'
assert kv.get('i') == 123
assert kv.get('d') == datetime.datetime.fromtimestamp(12325)
assert kv.get('n') == decimal.Decimal('1234.56')
assert kv.get('ss') == set(range(10))
assert kv.get('o') == dict(d=decimal.Decimal('1234.58'), 
                           n=datetime.datetime.fromtimestamp(12325))

Listing values

keys() and items() methods return a generator yielding the values for efficient stream processing.

The returned data is sorted ascending (by default) based on the keys

assert list(kv.keys()) == ['d', 'i', 'n', 'o', 's', 'ss']
assert list(kv.items()) == [
  ('d', datetime.datetime.fromtimestamp(12325)), 
  ('i', 123), 
  ('n', decimal.Decimal('1234.56')), 
  ('o', {'d': decimal.Decimal('1234.58'), 
         'n': datetime.datetime.fromtimestamp(12325)}), 
  ('s', 'value'), 
  ('ss', {0, 1, 2, 3, 4, 5, 6, 7, 8, 9})
]

Set the reverse argument to True for the keys() and items() methods to sort in descending order.

Bulk inserting data

The SQLite DB backend can be very slow when bulk inserting data. You can use the insert method to insert efficiently in bulk.

kv.insert(((str(i), ':{}'.format(i)) for i in range(50000)))

The batch size is 1000 by default, you should modify it depending on the size of your data and available memory.

kv.insert(((str(i), ':{}'.format(i)) for i in range(50000)), batch_size=40000)

If you are inserting data from a generator and need to use the inserted data, use insert_generator method:

for key, value in kv.insert_generator(((str(i), ':{}'.format(i)) for i in range(50)), batch_size=10):
    print(key, value)

Installing leveldb

On Debian based Linux:

$ apt-get install libleveldb-dev libleveldb1

On Alpine based Linux:

$ apk --repository http://dl-3.alpinelinux.org/alpine/edge/testing/ --update add leveldb leveldb-dev

On OS X:

$ brew install leveldb

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

kvfile-1.1.2.tar.gz (11.7 kB view details)

Uploaded Source

Built Distribution

kvfile-1.1.2-py2.py3-none-any.whl (9.4 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file kvfile-1.1.2.tar.gz.

File metadata

  • Download URL: kvfile-1.1.2.tar.gz
  • Upload date:
  • Size: 11.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.12.1

File hashes

Hashes for kvfile-1.1.2.tar.gz
Algorithm Hash digest
SHA256 5af57c40aa0f26c60b715360d08b6e46d13f3771466f40444133c752f34a581d
MD5 ccda6adc1d81891ef53f0d8d2efc57a4
BLAKE2b-256 9f5c605d59e80fcc7ea9a520bdd8a5713e2de9cc2952a0ec2cce92135a9fd885

See more details on using hashes here.

File details

Details for the file kvfile-1.1.2-py2.py3-none-any.whl.

File metadata

  • Download URL: kvfile-1.1.2-py2.py3-none-any.whl
  • Upload date:
  • Size: 9.4 kB
  • Tags: Python 2, Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.12.1

File hashes

Hashes for kvfile-1.1.2-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 ac40a79b2df1b7b3806c336827327d4ff52d3df975dd44b4543b443bf686fed6
MD5 5ad58490efd8ac0dcc4af6a434902dd7
BLAKE2b-256 40fb80e3f56d31730eee744418451a2baa12903b33b80823777a03875dcb7c24

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page