A wrapper for select object operations using boto3 for Ceph which confirms integrity of reads and heals intermittent connectivity.
Project description
A wrapper for the following boto3 s3 client operations with connection retry and checksum verification:
get_object
head_object
upload_fileobj
copy_object
delete_object
Features
Retry connection failures
Confirm checksum of uploaded and retrieved objects
Move object
Constrained interface to support simple CRUD operations for objects in existing buckets
Installation
pip install object-clerkExamples
Initialize
clerk = ObjectClerk(host=127.0.0.1, port=8080, access_key=12342, secret_key=12342, retry_delay=1, retry_backoff=1, retry_jitter=(1, 3), retry_max_delay=5, retry_tries=3, use_ssl=False)'Get Object
# with checksum verified
bytes_response = clerk.get_object("bucket", "object_key")
# without checksum verified
bytes_response = clerk.get_object("bucket", "object_key", verify_checksum=False)Delete Object
clerk.delete_object("bucket", "object_key")Get Object Info
dict_response = clerk.get_object_info("bucket", "object_key")Copy Object
# with checksum verified
clerk.copy_object(
"source_bucket",
"source_object_key",
"destination_bucket",
"destination_object_key",
)
# without checksum verified
clerk.copy_object(
"source_bucket",
"source_object_key",
"destination_bucket",
"destination_object_key",
verify_checksum=False
)Upload Object
# with checksum verified
with open("file", mode='rb') as f:
clerk.upload_object(f, "bucket", "object_key")
# without checksum verified
with open("file", mode='rb') as f:
clerk.upload_object(f, "bucket", "object_key", verify_checksum=False)Move Object
# with checksum verified
clerk.move_object(
"source_bucket",
"source_object_key",
"destination_bucket",
"destination_object_key",
)
# without checksum verified
clerk.move_object(
"source_bucket",
"source_object_key",
"destination_bucket",
"destination_object_key",
verify_checksum=False
)Test
git clone git@bitbucket.org:swiant/object_store_wrapper.git
pip install -e .
export HOST=<host>
export PORT=<port>
export ACCESS_KEY=<access_key>
export SECRET_KEY=<secret_key>
pytest -v object_clerkRelease history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
object-clerk-0.0.4.tar.gz
(13.3 kB
view details)
File details
Details for the file object-clerk-0.0.4.tar.gz.
File metadata
- Download URL: object-clerk-0.0.4.tar.gz
- Upload date:
- Size: 13.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.42.1 CPython/3.7.3
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | baffff3d659b57973123a6e4178439b2d0f7d436f080c68f2472f752127a268a |
|
| MD5 | 9f083f9eb8d20faca60dfa05d76b749c |
|
| BLAKE2b-256 | 2a9bad5b8499d96cf3a51685be67e458605fe9d63ab85978499567465b897c90 |
