A wrapper for select object operations using boto3 for Ceph which confirms integrity of reads and heals intermittent connectivity.
Project description
A wrapper for the following boto3 s3 client operations with connection retry and checksum verification:
get_object
head_object
upload_fileobj
copy_object
delete_object
list_objects_v2
Features
Retry connection failures
Confirm checksum of uploaded and retrieved objects
Move object
Constrained interface to support simple CRUD operations for objects in existing buckets
Installation
pip install object-clerkExamples
Initialize
clerk = ObjectClerk(host=127.0.0.1, port=8080, access_key=12342, secret_key=12342, retry_delay=1, retry_backoff=1, retry_jitter=(1, 3), retry_max_delay=5, retry_tries=3, use_ssl=False)'Get Object
# with checksum verified
bytes_response = clerk.get_object("bucket", "object_key")
# without checksum verified
bytes_response = clerk.get_object("bucket", "object_key", verify_checksum=False)Delete Object
clerk.delete_object("bucket", "object_key")Get Object Info
dict_response = clerk.get_object_info("bucket", "object_key")Copy Object
# with checksum verified
clerk.copy_object(
"source_bucket",
"source_object_key",
"destination_bucket",
"destination_object_key",
)
# without checksum verified
clerk.copy_object(
"source_bucket",
"source_object_key",
"destination_bucket",
"destination_object_key",
verify_checksum=False
)Upload Object
# with checksum verified
with open("file", mode='rb') as f:
clerk.upload_object(f, "bucket", "object_key")
# without checksum verified
with open("file", mode='rb') as f:
clerk.upload_object(f, "bucket", "object_key", verify_checksum=False)Move Object
# with checksum verified
clerk.move_object(
"source_bucket",
"source_object_key",
"destination_bucket",
"destination_object_key",
)
# without checksum verified
clerk.move_object(
"source_bucket",
"source_object_key",
"destination_bucket",
"destination_object_key",
verify_checksum=False
)List Object
clerk.list_objects(
"bucket_name",
1000
)Test
git clone git@bitbucket.org:swiant/object_store_wrapper.git
pip install -e .
export HOST=<host>
export PORT=<port>
export ACCESS_KEY=<access_key>
export SECRET_KEY=<secret_key>
pytest -v object_clerkRelease history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
object-clerk-0.0.8.tar.gz
(13.9 kB
view details)
File details
Details for the file object-clerk-0.0.8.tar.gz.
File metadata
- Download URL: object-clerk-0.0.8.tar.gz
- Upload date:
- Size: 13.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.43.0 CPython/3.7.3
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | a779e35f23ed8884ac3887170e273d0284b428aabc0bd1ec9460f14ae908f928 |
|
| MD5 | 98fc711e31279eed4df31e9f4e0e2bee |
|
| BLAKE2b-256 | 20cc4702a4baedbfe9cf19a6979a122f0fce89224ed734738d0b7c02c5d4467f |
