Simple File-based KV-Store
Project description
kvfile
A simple Key-Value store that's file based - so can accommodate large data sets with a small memory footprint.
Internally will use the faster leveldb
as a storage backend or sqlite3
as fallback if leveldb
is not available.
The Basics
The API should feel familiar to anyone working with Python.
It exposes get
, keys
and items
for reading from the DB, and set
for setting a value in the DB.
Initializing
import datetime
import decimal
from kvfile import KVFile
kv = KVFile()
Setting values
kv.set('s', 'value')
kv.set('i', 123)
kv.set('d', datetime.datetime.fromtimestamp(12325))
kv.set('n', decimal.Decimal('1234.56'))
kv.set('ss', set(range(10)))
kv.set('o', dict(d=decimal.Decimal('1234.58'),
n=datetime.datetime.fromtimestamp(12325)))
Getting values
assert kv.get('s') == 'value'
assert kv.get('i') == 123
assert kv.get('d') == datetime.datetime.fromtimestamp(12325)
assert kv.get('n') == decimal.Decimal('1234.56')
assert kv.get('ss') == set(range(10))
assert kv.get('o') == dict(d=decimal.Decimal('1234.58'),
n=datetime.datetime.fromtimestamp(12325))
Listing values
keys()
and items()
methods return a generator yielding the values for efficient stream processing.
The returned data is sorted ascending (by default) based on the keys
assert list(kv.keys()) == ['d', 'i', 'n', 'o', 's', 'ss']
assert list(kv.items()) == [
('d', datetime.datetime.fromtimestamp(12325)),
('i', 123),
('n', decimal.Decimal('1234.56')),
('o', {'d': decimal.Decimal('1234.58'),
'n': datetime.datetime.fromtimestamp(12325)}),
('s', 'value'),
('ss', {0, 1, 2, 3, 4, 5, 6, 7, 8, 9})
]
Set the reverse
argument to True for the keys()
and items()
methods to sort in descending order.
Bulk inserting data
The SQLite DB backend can be very slow when bulk inserting data. You can use the insert method to insert efficiently in bulk.
kv.insert(((str(i), ':{}'.format(i)) for i in range(50000)))
The batch size is 1000 by default, you should modify it depending on the size of your data and available memory.
kv.insert(((str(i), ':{}'.format(i)) for i in range(50000)), batch_size=40000)
If you are inserting data from a generator and need to use the inserted data, use insert_generator
method:
for key, value in kv.insert_generator(((str(i), ':{}'.format(i)) for i in range(50)), batch_size=10):
print(key, value)
Installing leveldb
On Debian based Linux:
$ apt-get install libleveldb-dev libleveldb1
On Alpine based Linux:
$ apk --repository http://dl-3.alpinelinux.org/alpine/edge/testing/ --update add leveldb leveldb-dev
On OS X:
$ brew install leveldb
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file kvfile-1.1.2.tar.gz
.
File metadata
- Download URL: kvfile-1.1.2.tar.gz
- Upload date:
- Size: 11.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.12.1
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5af57c40aa0f26c60b715360d08b6e46d13f3771466f40444133c752f34a581d |
|
MD5 | ccda6adc1d81891ef53f0d8d2efc57a4 |
|
BLAKE2b-256 | 9f5c605d59e80fcc7ea9a520bdd8a5713e2de9cc2952a0ec2cce92135a9fd885 |
File details
Details for the file kvfile-1.1.2-py2.py3-none-any.whl
.
File metadata
- Download URL: kvfile-1.1.2-py2.py3-none-any.whl
- Upload date:
- Size: 9.4 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.12.1
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | ac40a79b2df1b7b3806c336827327d4ff52d3df975dd44b4543b443bf686fed6 |
|
MD5 | 5ad58490efd8ac0dcc4af6a434902dd7 |
|
BLAKE2b-256 | 40fb80e3f56d31730eee744418451a2baa12903b33b80823777a03875dcb7c24 |