Python bindings for SQLite's LSM key/value engine

These details have not been verified by PyPI

Project links

Project description

lsm

Fast Python bindings for SQLite's LSM key/value store. The LSM storage engine was initially written as part of the experimental SQLite4 rewrite (now abandoned). More recently, the LSM source code was moved into the SQLite3 source tree and has seen some improvements and fixes. This project uses the LSM code from the SQLite3 source tree.

Features:

Embedded zero-conf database.
Keys support in-order traversal using cursors.
Transactional (including nested transactions).
Single writer/multiple reader MVCC based transactional concurrency model.
On-disk database stored in a single file.
Data is durable in the face of application or power failure.
Thread-safe.
Releases GIL for read and write operations (each connection has own mutex)
Page compression (lz4 or zstd)
Zero dependency static library
Python 3.x.

Limitations:

The source for Python lsm is hosted on GitHub.

If you encounter any bugs in the library, please open an issue, including a description of the bug and any related traceback.

Quick-start

Below is a sample interactive console session designed to show some of the basic features and functionality of the lsm Python library.

To begin, instantiate a LSM object, specifying a path to a database file.

>>> from lsm import LSM
>>> db = LSM('test.ldb')
>>> db.open()
>>> print(db)
<LSM at "/tmp/test.ldb" as 0x10951e450>

More pythonic variant is using context manager:

>>> from lsm import LSM
>>> with LSM("/tmp/test.ldb") as db:
...     print(db)
<LSM at "/tmp/test.ldb" as 0x10951e450>

Binary/string mode

You should select mode for opening the database with binary: bool = True argument.

For example when you want to store strings just pass binary=False:

>>> from lsm import LSM
>>> with LSM("/tmp/test.ldb", binary=False) as db:
...    db['foo'] = 'bar'   # must be str for keys and values
...    print(db['foo'])
bar

Otherwise, you must pass keys and values ad bytes (default behaviour):

>>> from lsm import LSM
>>> with LSM("/tmp/test.ldb") as db:
...    db[b'foo'] = b'bar'   # must be bytes for keys and values
...    print(db[b'foo'])
b'bar'

Key/Value Features

lsm is a key/value store, and has a dictionary-like API:

>>> from lsm import LSM
>>> with LSM("/tmp/test.ldb", binary=False) as db:
...    db['foo'] = 'bar'
...    print(db['foo'])
bar

Database apply changes as soon as possible:

>>> from lsm import LSM
>>> db = LSM("/tmp/test.ldb", binary=False)
>>> db.open()
True
>>> for i in range(4):
...     db[f'k{i}'] = str(i)
...
>>> 'k3' in db
True
>>> 'k4' in db
False
>>> del db['k3']
>>> db['k3']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: "Key 'k3' was not found"

By default when you attempt to look up a key, lsm will search for an exact match. You can also search for the closest key, if the specific key you are searching for does not exist:

>>> from lsm import, LSM, SEEK_LE, SEEK_GE
>>> db = LSM("/tmp/test.ldb", binary=False)
>>> db.open()
True
>>> db['k1xx', SEEK_LE]  # Here we will match "k1".
'1'
>>> db['k1xx', SEEK_GE]  # Here we will match "k2".
'2'

LSM supports other common dictionary methods such as:

keys()
values()
items()
update()

Slices and Iteration

The database can be iterated through directly, or sliced. When you are slicing the database the start and end keys need not exist -- lsm will find the closest key (details can be found in the LSM.fetch_range() documentation).

>>> [item for item in db.items()]
[('foo', 'bar'), ('k0', '0'), ('k1', '1'), ('k2', '2')]

>>> db['k0':'k99']
<lsm_slice object at 0x10d4f3500>

>>> list(db['k0':'k99'])
[('k0', '0'), ('k1', '1'), ('k2', '2')]

You can use open-ended slices. If the lower- or upper-bound is outside the range of keys an empty list is returned.

>>> list(db['k0':])
[('k0', '0'), ('k1', '1'), ('k2', '2')]

>>> list(db[:'k1'])
[('foo', 'bar'), ('k0', '0'), ('k1', '1')]

>>> list(db[:'aaa'])
[]

To retrieve keys in reverse order or stepping over more then one item, simply use a third slice argument as usual. Negative step value means reverse order, but first and second arguments must be ordinary ordered.

>>> list(db['k0':'k99':2])
[('k0', '0'), ('k2', '2')]

>>> list(db['k0'::-1])
[('k2', '2'), ('k1', '1'), ('k0', '0')]

>>> list(db['k0'::-2])
[('k2', '2'), ('k0', '0')]


>>> list(db['k0'::3])
[('k0', '0')]

You can also delete slices of keys, but note that the delete will not include the keys themselves:

>>> del db['k0':'k99']

>>> list(db)  # Note that 'k0' still exists.
[('foo', 'bar'), ('k0', '0')]

Cursors

While slicing may cover most use-cases, for finer-grained control you can use cursors for traversing records.

>>> with db.cursor() as cursor:
...     for key, value in cursor:
...         print(key, '=>', value)
...
foo => bar
k0 => 0

>>> db.update({'k1': '1', 'k2': '2', 'k3': '3', 'foo': 'bar'})

>>> with db.cursor() as cursor:
...     cursor.first()
...     print(cursor.key())
...     cursor.last()
...     print(cursor.key())
...     cursor.previous()
...     print(cursor.key())
...
foo
k3
k2

>>> with db.cursor() as cursor:
...     cursor.seek('k0', SEEK_GE)
...     print(list(cursor.fetch_until('k99')))
...
[('k0', '0'), ('k1', '1'), ('k2', '2'), ('k3', '3')]

It is very important to close a cursor when you are through using it. For this reason, it is recommended you use the LSM.cursor() context-manager, which ensures the cursor is closed properly.

Transactions

lsm supports nested transactions. The simplest way to use transactions is with the LSM.transaction() method, which doubles as a context-manager or decorator.

>>> with db.transaction() as txn:
...     db['k1'] = '1-mod'
...     with db.transaction() as txn2:
...         db['k2'] = '2-mod'
...         txn2.rollback()
...
True
>>> print(db['k1'], db['k2'])
1-mod 2

You can commit or roll-back transactions part-way through a wrapped block:

>>> with db.transaction() as txn:
...    db['k1'] = 'outer txn'
...    txn.commit()  # The write is preserved.
...
...    db['k1'] = 'outer txn-2'
...    with db.transaction() as txn2:
...        db['k1'] = 'inner-txn'  # This is commited after the block ends.
...    print(db['k1']  # Prints "inner-txn".)
...    txn.rollback()  # Rolls back both the changes from txn2 and the preceding write.
...    print(db['k1'])
...
1              <- Return value from call to commit().
inner-txn      <- Printed after end of txn2.
True           <- Return value of call to rollback().
outer txn      <- Printed after rollback.

If you like, you can also explicitly call LSM.begin(), LSM.commit(), and LSM.rollback().

>>> db.begin()
>>> db['foo'] = 'baze'
>>> print(db['foo'])
baze
>>> db.rollback()
True
>>> print(db['foo'])
bar

Thanks to

@coleifer - this project was inspired by coleifer/python-lsm-db.

Project details

These details have not been verified by PyPI

Project links

Release history Release notifications | RSS feed

0.5.4

Apr 5, 2023

0.5.3

Mar 27, 2023

0.5.2

Mar 2, 2023

0.4.9

Jan 29, 2023

0.4.8

Jan 24, 2023

0.4.7

Jan 24, 2023

0.4.6

Dec 22, 2022

0.4.5

Dec 22, 2022

This version

0.4.4

Nov 19, 2021

0.4.3

Nov 18, 2021

0.4.2

Nov 16, 2021

0.4.1

Nov 16, 2021

0.4.0

Nov 16, 2021

0.3.12

Nov 16, 2021

0.3.11

Nov 16, 2021

0.3.10

Nov 16, 2021

0.3.9

Nov 16, 2021

0.3.8

Aug 11, 2021

0.3.7

Jul 12, 2021

0.3.6

Jun 23, 2021

0.3.5

Jun 21, 2021

0.3.2

Jun 11, 2021

0.3.1

Jun 9, 2021

0.3.0

Jun 9, 2021

0.2.4

May 31, 2021

0.2.1

May 30, 2021

0.2.0

May 30, 2021

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

lsm-0.4.4.tar.gz (799.0 kB view details)

Uploaded Nov 19, 2021 Source

Built Distributions

lsm-0.4.4-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.1 MB view details)

Uploaded Nov 9, 2022 CPython 3.11 manylinux: glibc 2.17+ ARM64

lsm-0.4.4-cp310-cp310-win_amd64.whl (1.2 MB view details)

Uploaded Nov 19, 2021 CPython 3.10 Windows x86-64

lsm-0.4.4-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.1 MB view details)

Uploaded Nov 9, 2022 CPython 3.10 manylinux: glibc 2.17+ ARM64

lsm-0.4.4-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.4 MB view details)

Uploaded Nov 19, 2021 CPython 3.10 manylinux: glibc 2.12+ x86-64 manylinux: glibc 2.5+ x86-64

lsm-0.4.4-cp310-cp310-macosx_10_14_x86_64.whl (1.5 MB view details)

Uploaded Nov 19, 2021 CPython 3.10 macOS 10.14+ x86-64

lsm-0.4.4-cp310-cp310-macosx_10_9_universal2.whl (2.7 MB view details)

Uploaded Feb 21, 2022 CPython 3.10 macOS 10.9+ universal2 (ARM64, x86-64)

lsm-0.4.4-cp39-cp39-win_amd64.whl (1.2 MB view details)

Uploaded Nov 19, 2021 CPython 3.9 Windows x86-64

lsm-0.4.4-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.1 MB view details)

Uploaded Nov 9, 2022 CPython 3.9 manylinux: glibc 2.17+ ARM64

lsm-0.4.4-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.4 MB view details)

Uploaded Nov 19, 2021 CPython 3.9 manylinux: glibc 2.12+ x86-64 manylinux: glibc 2.5+ x86-64

lsm-0.4.4-cp39-cp39-macosx_10_14_x86_64.whl (1.5 MB view details)

Uploaded Nov 19, 2021 CPython 3.9 macOS 10.14+ x86-64

lsm-0.4.4-cp39-cp39-macosx_10_9_universal2.whl (2.7 MB view details)

Uploaded Feb 21, 2022 CPython 3.9 macOS 10.9+ universal2 (ARM64, x86-64)

lsm-0.4.4-cp38-cp38-win_amd64.whl (1.2 MB view details)

Uploaded Nov 19, 2021 CPython 3.8 Windows x86-64

lsm-0.4.4-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.1 MB view details)

Uploaded Nov 9, 2022 CPython 3.8 manylinux: glibc 2.17+ ARM64

lsm-0.4.4-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.4 MB view details)

Uploaded Nov 19, 2021 CPython 3.8 manylinux: glibc 2.12+ x86-64 manylinux: glibc 2.5+ x86-64

lsm-0.4.4-cp38-cp38-macosx_11_0_universal2.whl (2.7 MB view details)

Uploaded Feb 21, 2022 CPython 3.8 macOS 11.0+ universal2 (ARM64, x86-64)

lsm-0.4.4-cp38-cp38-macosx_10_14_x86_64.whl (1.5 MB view details)

Uploaded Nov 19, 2021 CPython 3.8 macOS 10.14+ x86-64

lsm-0.4.4-cp37-cp37m-win_amd64.whl (1.2 MB view details)

Uploaded Nov 19, 2021 CPython 3.7m Windows x86-64

lsm-0.4.4-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.1 MB view details)

Uploaded Nov 9, 2022 CPython 3.7m manylinux: glibc 2.17+ ARM64

lsm-0.4.4-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.4 MB view details)

Uploaded Nov 19, 2021 CPython 3.7m manylinux: glibc 2.12+ x86-64 manylinux: glibc 2.5+ x86-64

lsm-0.4.4-cp37-cp37m-macosx_10_14_x86_64.whl (1.5 MB view details)

Uploaded Nov 19, 2021 CPython 3.7m macOS 10.14+ x86-64

lsm-0.4.4-cp36-cp36m-win_amd64.whl (1.2 MB view details)

Uploaded Nov 19, 2021 CPython 3.6m Windows x86-64

lsm-0.4.4-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.4 MB view details)

Uploaded Nov 19, 2021 CPython 3.6m manylinux: glibc 2.12+ x86-64 manylinux: glibc 2.5+ x86-64

lsm-0.4.4-cp36-cp36m-macosx_10_14_x86_64.whl (1.5 MB view details)

Uploaded Nov 19, 2021 CPython 3.6m macOS 10.14+ x86-64

File details

Details for the file lsm-0.4.4.tar.gz.

File metadata

Download URL: lsm-0.4.4.tar.gz
Upload date: Nov 19, 2021
Size: 799.0 kB
Tags: Source
Uploaded using Trusted Publishing? No
Uploaded via: twine/3.6.0 importlib_metadata/4.8.2 pkginfo/1.8.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.9.8

File hashes

Hashes for lsm-0.4.4.tar.gz
Algorithm	Hash digest
SHA256	`e25fd183d043c73e7b7329b0cdd378b8773eb2b9cc5ea282c0f09a33193a91e6`
MD5	`061841e1f7658f3a738d8ab582255b85`
BLAKE2b-256	`df842864cb4641d7dd982eaf6b376dd7123a5597117780209f5ce89dca131fb5`

Algorithm	Hash digest
SHA256	`aef62f32272591c984aafaa15a32356907db319949939e4ac52467767a308e71`
MD5	`efcfd7daa6393c565c0e7a993f5d8ced`
BLAKE2b-256	`3718866e48ce27398e2e601119fee9f44399dd626ff3b7d0b262bf988b659c9e`

Algorithm	Hash digest
SHA256	`c681d2d8aa18592dd3323af195c6cc95c6c29f550e6dfbf8db95c8c8873b8f9b`
MD5	`eae2668425c31538b8bbde31ab03ffa6`
BLAKE2b-256	`41f96868620bff6d69f56dec1f97cdbfb5cefcc9cca21304d841706fb482fa8a`

Algorithm	Hash digest
SHA256	`7ca5905f5ed666bc2821f980d8464fc6a99e9cbccff03b16def740a06060a9a3`
MD5	`871a374f4ee43ca4747a7f4d1e370d0a`
BLAKE2b-256	`6ed22a5b20051acad211e0143411ff93495c291cd00ad85ee83818eaff9ed9a1`

Algorithm	Hash digest
SHA256	`b816fc94ad7a6a5855cdc080c6e35021a1ae485b6cc1312f28a1bd8fbecbea4d`
MD5	`a2b63a155bbeebfcc1eca7e62e4b48c0`
BLAKE2b-256	`e7b8761f5f10073397259fd987edea6df9bb83f1d89c5ddfbac4419becdd7540`

Algorithm	Hash digest
SHA256	`a6a57ae4b4afa482933a7af5d670991ad1dbb2f5788bf9d55410ab3219d24f15`
MD5	`fb367109cd7d1b7bed525821f0f23884`
BLAKE2b-256	`884cd51525e8086ef607e0f8bc8d68fe0d210b5a5f351ab3b53e3229a3b1c4fb`

Algorithm	Hash digest
SHA256	`72ef776822c30a6a4ba49e09e45cacb4a00efd70a4cbb040278237e4e2b0cf02`
MD5	`0a71896ab4d48bbcc3d5b780faed93b4`
BLAKE2b-256	`42c08265d8ceaac4438941084619043606d1471715853331897ffb6c3bbd875c`

Algorithm	Hash digest
SHA256	`4446f0136df76ee4f14e766bc857a5ea96831da1e9f238123dc340e83a9d37c8`
MD5	`b2e03650cec5489c60be959842eafc9c`
BLAKE2b-256	`4437452d307ae13f14f6669f40b02498857f9b926d28deda50bc61c76e474e3d`

Algorithm	Hash digest
SHA256	`af5dd1363d89d30d8e35b54efdbc887d10a9584fd94d31831bbfcfcec0e16738`
MD5	`68b733498c1b13fe851829e4f9d90a27`
BLAKE2b-256	`fea909f59a9d6f392a8defc44b20b5e4fa80040c5f95651865f05aadbfc00ad0`

Algorithm	Hash digest
SHA256	`e2851c788ac5ba36d35eba51781fc2badbed8994dba2f9a10bd38a30f11d2ee4`
MD5	`aa72af67a5d7758fdcaa9a0cd95fa40c`
BLAKE2b-256	`17cdf54d5dc0db139da881cd790d29294194de38c6ad63641363f0c5f39f54a7`

Algorithm	Hash digest
SHA256	`749a0a43ea1bbdb0927191cabad8dfe6ebd308edc536b0d8d72c9ad31384000d`
MD5	`324d6f5648aa47ef2d158bd349e8f6c4`
BLAKE2b-256	`abd749ba3ce5257d5c3798836ea02f81828fcc3dff58ad03bacaca2f1c546cca`

Algorithm	Hash digest
SHA256	`3daa194f580bae14eff27afe7d275c4c6a80b8b8a540f716a0efec894b123d54`
MD5	`9644a0958ab9e06d7808765960a86b96`
BLAKE2b-256	`21ffb92a878aed892fe9733196f8edcde4ecd9cc5a2a2e6c75e9da4f560da572`

Algorithm	Hash digest
SHA256	`aef3d59b85e5bb9f335066c673984b710a0dc9415379076f655029800cf9eb30`
MD5	`5ead4362358d2d43c5d735149d09dda7`
BLAKE2b-256	`e477d430f005d9dc9795d9e47be2c29b0cf77aa3493d4f167bc68fd9e228aacb`

Algorithm	Hash digest
SHA256	`023a56792770070eec970e31ae063a81e7c6809587912472bde939e51d415390`
MD5	`7309c9e12aec6dc0ce94a017f3399684`
BLAKE2b-256	`220f671c29af28798c57dfbdb926b23b13b102b0199d705e891b6709486a1f71`

Algorithm	Hash digest
SHA256	`b6042d7300f17c5b79885f236e706ed2d2b0df4175cd8cc589decb1812b46c0b`
MD5	`d4f865850795988f105b4b48a5fc2147`
BLAKE2b-256	`33b5f69dc6a0bcc750be00a941343c0fdee8643fbb4d46691fd5d40aec69c0a5`

Algorithm	Hash digest
SHA256	`0b7d9be11f972e618416108a8b0f292e874157da801fd597417ea05b4b5fd07c`
MD5	`d2226f02e730883cf9992bcc9dd69e16`
BLAKE2b-256	`880fe96f2ee61a19a8be3eb8059c4eb2f6711d2e35c983a247be9d636040babd`

Algorithm	Hash digest
SHA256	`20646020cca15c9eb9a19efe00c4a0e06c2b627c6f58174649c3a7eaa4206920`
MD5	`a4071fb01735ce608bc210578c8004ad`
BLAKE2b-256	`a85e39cc837e5ea547b7a241fefe6f85423b0c62f46395145ea9670d59a8cbee`

Algorithm	Hash digest
SHA256	`f69690e7a93f36478cbc870bc2927ef240da2ed7a68cd67ef294441e4536342b`
MD5	`04d1ceef20c5549c1d616d9cbaee36c1`
BLAKE2b-256	`0e9aab2b0a8ce5237da029d0a35b47f945cba204343d70eae8da257529c1d16f`

Algorithm	Hash digest
SHA256	`c4d7f309f677e19340c719b01667e77dd2d15b814b34f00dd9199462e3a8bfc2`
MD5	`252be184dacca436a54ddecb4873dafe`
BLAKE2b-256	`27689f3db0e92290bbae90eea51dae8fce743b656cda03d417607d328f493bcf`

Algorithm	Hash digest
SHA256	`d1e2e75b3739da4f5006c0bc592748b1ef3c2b4ef22fdcf16ec0d313951ebfb8`
MD5	`7b6f7e617d23f0b691eec155225f89a4`
BLAKE2b-256	`5eeddcac5a6f05c447557d2196173fd637cd52c0b9efb4303a1e8073f83e86d5`

Algorithm	Hash digest
SHA256	`671ff84f1fba7fd75ded28acf0636f6d026302278b8a593fd2377ac5a8250e59`
MD5	`bff8270f0ff4af19bb39550c2a23f9d3`
BLAKE2b-256	`377f97ff968680da09d4f4eca6aab30af6cfa49a02cd05544ece15e742fafbbb`

lsm 0.4.4

Navigation

Verified details

Maintainers

Unverified details

Project links

Meta

Classifiers

Project description

lsm

Quick-start

Binary/string mode

Key/Value Features

Slices and Iteration

Cursors

Transactions

Thanks to

Project details

Verified details

Maintainers

Unverified details

Project links

Meta

Classifiers

Release history Release notifications | RSS feed

Download files

Source Distribution

Built Distributions

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

Algorithm	Hash digest
SHA256	`08aca541be77f56eacd33f170c11d6b3e3bd6189114d799ecb537f89d71dfbdd`
MD5	`cb0330b36d90b561be6dde1ade53b01e`
BLAKE2b-256	`64075c324afc84082bf2020caf20339b2c9d6cf73fdb200145c13d5dc990d132`

Algorithm	Hash digest
SHA256	`57290cb2180893644b5512ac695ce5950d90fcb372440b3ff0080d9d26ed0f21`
MD5	`19faa88ccfdc114892b5ebc95fae8551`
BLAKE2b-256	`7cd3c845331942ab997f625847b92c006064ad6089ed7aa4f9c01c7eb524d702`

Algorithm	Hash digest
SHA256	`8f96529f40bac6b2d24d1a6099c38e2b962304b37e241cd369d9593241e01ce7`
MD5	`e50c769f6802187d5d34df03a8787ed7`
BLAKE2b-256	`dfd4d1c60dc87df3905dc8dc8a83ec6eb9332b03add9953991e08033436268b5`