A wrapper for select object operations using boto3 on Ceph which confirms integrity of reads and heals intermittent connectivity.
Project description
A wrapper for the following boto3 s3 client operations below the multipart upload threshold of 5 GB:
get_object
head_object
upload_fileobj
copy_object
delete_object
Features
Retry connection failures
Confirm checksum of retrieved objects
Constrained interface to support simple CRUD operations for objects in existing buckets
Installation
pip install object-clerkExamples
Initialize
clerk = ObjectClerk(host=127.0.0.1, port=8080, access_key=12342, secret_key=12342, retry_delay=1, retry_backoff=1, retry_jitter=(1, 3), retry_max_delay=5, retry_tries=3, use_ssl=False)'Get Object
# with checksum verified
bytes_response = clerk.get_object("bucket", "object_key")
# without checksum verified
bytes_response = clerk.get_object("bucket", "object_key", verify_checksum=False)Delete Object
clerk.delete_object("bucket", "object_key")Get Object Info
dict_response = clerk.get_object_info("bucket", "object_key")Copy Object
# with checksum verified
clerk.copy_object(
"source_bucket",
"source_object_key",
"destination_bucket",
"destination_object_key",
)
# without checksum verified
clerk.copy_object(
"source_bucket",
"source_object_key",
"destination_bucket",
"destination_object_key",
verify_checksum=False
)Upload Object
# with checksum verified
with open("file", mode='rb') as f:
clerk.upload_object(f, "bucket", "object_key")
# without checksum verified
with open("file", mode='rb') as f:
clerk.upload_object(f, "bucket", "object_key", verify_checksum=False)Test
git clone git@bitbucket.org:swiant/object_store_wrapper.git
pip install -e .
export HOST=<host>
export PORT=<port>
export ACCESS_KEY=<access_key>
export SECRET_KEY=<secret_key>
pytest -v object_clerkRelease history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
object-clerk-0.0.1.tar.gz
(10.7 kB
view details)
File details
Details for the file object-clerk-0.0.1.tar.gz.
File metadata
- Download URL: object-clerk-0.0.1.tar.gz
- Upload date:
- Size: 10.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.42.1 CPython/3.7.3
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 6affabed0da39e7234aad8a4c26f2669207fc23116bbb79f1573b19d76d10932 |
|
| MD5 | 359e5560d6be9ab4a8f81b484996e2f9 |
|
| BLAKE2b-256 | 6871ad492427d8beb0d460c93a41c530a39ebd97f28cc8a7a630701c93ca3f1c |
