Redis for Humans.
Project description
Pottery: Redis for Humans
Redis is awesome, :grinning: but Redis commands are not always fun. :rage: Pottery is a Pythonic way to access Redis. If you know how to use Python dicts, then you already know how to use Pottery.
Installation
$ pip3 install pottery
Usage
First, set up your Redis client: :alien:
>>> from redis import Redis
>>> redis = Redis.from_url('redis://localhost:6379/')
That was the hardest part. :grimacing:
Dicts
Create a RedisDict:
>>> from pottery import RedisDict
>>> raj = RedisDict(redis=redis, key='raj')
Notice the two keyword arguments to RedisDict(): The first is your Redis
client. The second is the Redis key name for your dict. Other than that, you
can use your RedisDict the same way that you use any other Python dict:
>>> raj['hobby'] = 'music'
>>> raj['vegetarian'] = True
>>> raj
RedisDict{'hobby': 'music', 'vegetarian': True}
>>> len(raj)
2
>>> raj['vegetarian']
True
Sets
Create a RedisSet:
>>> from pottery import RedisSet
>>> edible = RedisSet(redis=redis, key='edible')
Again, notice the two keyword arguments to RedisSet(): The first is your
Redis client. The second is the Redis key name for your set. Other than that,
you can use your RedisSet the same way that you use any other Python set:
>>> edible.add('eggs')
>>> edible.extend({'beans', 'tofu', 'avocado'})
>>> edible
RedisSet{'tofu', 'avocado', 'eggs', 'beans'}
>>> len(edible)
4
>>> 'bacon' in edible
False
Lists
Create a RedisList:
>>> from pottery import RedisList
>>> lyrics = RedisList(redis=redis, key='lyrics')
Again, notice the two keyword arguments to RedisList(): The first is your
Redis client. The second is the Redis key name for your list. Other than
that, you can use your RedisList the same way that you use any other Python
list:
>>> lyrics.append('everything')
>>> lyrics.extend(['in', 'its', 'right', '...'])
>>> lyrics
RedisList['everything', 'in', 'its', 'right', '...']
>>> len(lyrics)
5
>>> lyrics[0]
'everything'
>>> lyrics[4] = 'place'
>>> lyrics
RedisList['everything', 'in', 'its', 'right', 'place']
NextId
NextId safely and reliably produces increasing IDs across threads, processes,
and even machines, without a single point of failure. Rationale and algorithm
description.
Instantiate an ID generator:
>>> from pottery import NextId
>>> user_ids = NextId(key='user-ids', masters={redis})
The key argument represents the sequence (so that you can have different
sequences for user IDs, comment IDs, etc.), and the masters argument
specifies your Redis masters across which to distribute ID generation (in
production, you should have 5 Redis masters). Now, whenever you need a user
ID, call next() on the ID generator:
>>> next(user_ids)
1
>>> next(user_ids)
2
>>> next(user_ids)
3
Two caveats:
- If many clients are generating IDs concurrently, then there may be “holes” in the sequence of IDs (e.g.: 1, 2, 6, 10, 11, 21, …).
- This algorithm scales to about 5,000 IDs per second (with 5 Redis masters). If you need IDs faster than that, then you may want to consider other techniques.
Redlock
Redlock is a safe and reliable lock to coordinate access to a resource shared
across threads, processes, and even machines, without a single point of
failure. Rationale and algorithm
description.
Redlock implements Python's excellent
threading.Lock
API as closely as is feasible. In other words, you can use Redlock the same
way that you use threading.Lock.
Instantiate a Redlock:
>>> from pottery import Redlock
>>> lock = Redlock(key='printer', masters={redis})
The key argument represents the resource, and the masters argument
specifies your Redis masters across which to distribute the lock (in
production, you should have 5 Redis masters). Now you can protect access to
your resource:
>>> lock.acquire()
>>> # Critical section - print stuff here.
>>> lock.release()
Or you can protect access to your resource inside a context manager:
>>> with lock:
... # Critical section - print stuff here.
Redlocks time out (by default, after 10 seconds). You should take care to
ensure that your critical section completes well within the timeout. The
reasons that Redlocks time out are to preserve
“liveness”
and to avoid deadlocks (in the event that a process dies inside a critical
section before it releases its lock).
>>> import time
>>> lock.acquire()
True
>>> bool(lock.locked())
True
>>> # Critical section - print stuff here.
>>> time.sleep(10)
>>> bool(lock.locked())
False
If 10 seconds isn't enough to complete executing your critical section, then you can specify your own timeout:
>>> lock = Redlock(key='printer', auto_release_time=15*1000)
>>> lock.acquire()
True
>>> bool(lock.locked())
True
>>> # Critical section - print stuff here.
>>> time.sleep(10)
>>> bool(lock.locked())
True
>>> time.sleep(5)
>>> bool(lock.locked())
False
ContextTimer
ContextTimer helps you easily and accurately measure elapsed time. Note that
ContextTimer measures wall (real-world) time, not CPU time; and that
elapsed() returns time in milliseconds.
You can use ContextTimer stand-alone…
>>> import time
>>> from pottery import ContextTimer
>>> timer = ContextTimer()
>>> timer.start()
>>> time.sleep(0.1)
>>> 100 <= timer.elapsed() < 200
True
>>> timer.stop()
>>> time.sleep(0.1)
>>> 100 <= timer.elapsed() < 200
True
…or as a context manager:
>>> tests = []
>>> with ContextTimer() as timer:
... time.sleep(0.1)
... tests.append(100 <= timer.elapsed() < 200)
>>> time.sleep(0.1)
>>> tests.append(100 <= timer.elapsed() < 200)
>>> tests
[True, True]
HyperLogLogs
HyperLogLogs are an interesting data structure that allow you to answer the question, “How many distinct elements have I seen?” but not the questions, “Have I seen this element before?” or “What are all of the elements that I’ve seen before?” So think of HyperLogLogs as Python sets that you can add elements to and get the length of, but that you can’t use to test element membership, iterate through, or get elements back out of.
Bloom filters
Bloom filters are a powerful data structure that help you to answer the questions, “Have I seen this element before?” and “How many distinct elements have I seen?” but not the question, “What are all of the elements that I’ve seen before?” So think of Bloom filters as Python sets that you can add elements to, use to test element membership, and get the length of, but that you can’t iterate through or get elements back out of.
Bloom filters are probabilistic, which means that they can sometimes generate false positives (as in, they may report that you’ve seen a particular element before even though you haven’t). But they will never generate false negatives (so every time that they report that you haven’t seen a particular element before, you really must never have seen it). You can tune your acceptable false positive probability, though at the expense of the storage size and the element insertion/lookup time of your Bloom filter.
Create a BloomFilter:
>>> from pottery import BloomFilter
>>> dilberts = BloomFilter(
... num_values=100,
... false_positives=0.01,
... redis=redis,
... key='dilberts',
... )
Here, num_values represents the number of elements that you expect to insert
into your BloomFilter, and false_positives represents your acceptable false
positive probability. Using these two parameters, BloomFilter automatically
computes its own storage size and number of times to run its hash functions on
element insertion/lookup such that it can guarantee a false positive rate at or
below what you can tolerate, given that you’re going to insert your
specified number of elements.
Insert an element into the BloomFilter:
>>> dilberts.add('rajiv')
Test for membership in the BloomFilter:
>>> 'rajiv' in dilberts
True
>>> 'raj' in dilberts
False
>>> 'dan' in dilberts
False
See how many elements we’ve inserted into the BloomFilter:
>>> len(dilberts)
1
Note that BloomFilter.__len__() is an approximation, so please don’t
rely on it for anything important like financial systems or cat gif websites.
Insert multiple elements into the BloomFilter:
>>> dilberts.update({'raj', 'dan'})
Remove all of the elements from the BloomFilter:
>>> dilberts.clear()
Contributing
Install prerequisites
- Install Xcode.
Obtain source code
- Clone the git repo:
$ git clone git@github.com:brainix/pottery.git$ cd pottery/- Install project-level dependencies:
$ make install
Run tests
- In one Terminal session:
$ cd pottery/$ redis-server- In a second Terminal session:
$ cd pottery/$ make test
make test runs all of the unit tests as well as the coverage test. However,
sometimes, when debugging, it can be useful to run an individual test module,
class, or method:
- In one Terminal session:
$ cd pottery/$ redis-server- In a second Terminal session:
- Run a test module with
$ make test tests=tests.test_dict - Run a test class with:
$ make test tests=tests.test_dict.DictTests - Run a test method with:
$ make test tests=tests.test_dict.DictTests.test_keyexistserror
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file pottery-1.0.0.tar.gz.
File metadata
- Download URL: pottery-1.0.0.tar.gz
- Upload date:
- Size: 30.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.25.0 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.48.2 CPython/3.9.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | d51bd8b23d3bfdf6574a5baab3f422d97fd70c585baa53407b4f0f1b2fa78733 |
|
| MD5 | 36e1bc8be264f7b1e43e9a86a4205bcf |
|
| BLAKE2b-256 | 908660a78140890467f0958634a8bcd7f1a51eb5e8f683112e9d1e82126d1fbb |
Provenance
File details
Details for the file pottery-1.0.0-py3-none-any.whl.
File metadata
- Download URL: pottery-1.0.0-py3-none-any.whl
- Upload date:
- Size: 39.5 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.25.0 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.48.2 CPython/3.9.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 541125686cee37f9f782eff85371ca59f0fecde76b20e433ff309596b3d5e890 |
|
| MD5 | 0dd5482ab712c1343b012f52d7b24ba3 |
|
| BLAKE2b-256 | a1c533831879061c6631e739d75eb3f2b59f6e00f8a88fba95f93c43b46c30b0 |
