object-clerk

A wrapper for select object operations using boto3 for Ceph which confirms integrity of reads and heals intermittent connectivity.

These details have not been verified by PyPI

Project links

Homepage

Project description

A wrapper for the following boto3 s3 client operations with connection retry and checksum verification:

get_object
head_object
upload_fileobj
copy_object
delete_object
list_objects_v2

Features

Retry connection failures
Confirm checksum of uploaded and retrieved objects
Move object
Constrained interface to support simple CRUD operations for objects in existing buckets

Configuration

The following environment variables configure the object clerk.

Variable	Description	Type	Default
MULTIPART_THRESHOLD	Threshold in bytes at which uploads are broken into multiple parts for upload. Impacts the checksum stored in the eTag	STR	524288000
S3_CLIENT_CONFIG	Boto Core Client Configuration https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html Defaults reflect capturing the retry of failed operations at a higher level.	JSON	`json {"connect_timeout": 60, "read_timeout": 60, "retries": {"max_attempts": 0}} `
S3_UPLOAD_CONFIG	Transfer Configuration applied to uploads https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html#boto3.s3.transfer.TransferConfig multipart_threshold and multipart_chunksize are set by the MULTIPART_THRESHOLD environment variable. Values for these keys in the S3_UPLOAD_CONFIG will be discarded.	JSON	`json {} `
S3_DOWNLOAD_CONFIG	Transfer Configuration applied to downloads https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html#boto3.s3.transfer.TransferConfig	JSON	`json {} `

Installation

pip install object-clerk

Examples

Initialize

clerk = ObjectClerk(host=127.0.0.1, port=8080, access_key=12342, secret_key=12342, retry_delay=1, retry_backoff=1, retry_jitter=(1, 3), retry_max_delay=5, retry_tries=3, use_ssl=False)'

Get Object

# with checksum verified

bytes_response = clerk.get_object("bucket", "object_key")

# without checksum verified

bytes_response = clerk.get_object("bucket", "object_key", verify_checksum=False)

Delete Object

clerk.delete_object("bucket", "object_key")

Get Object Info

dict_response = clerk.get_object_info("bucket", "object_key")

Copy Object

# with checksum verified

clerk.copy_object(
    "source_bucket",
    "source_object_key",
    "destination_bucket",
    "destination_object_key",
)

# without checksum verified

clerk.copy_object(
    "source_bucket",
    "source_object_key",
    "destination_bucket",
    "destination_object_key",
    verify_checksum=False
)

Upload Object

# with checksum verified

with open("file", mode='rb') as f:

    clerk.upload_object(f, "bucket", "object_key")

# without checksum verified

with open("file", mode='rb') as f:

    clerk.upload_object(f, "bucket", "object_key", verify_checksum=False)

Move Object

# with checksum verified

clerk.move_object(
    "source_bucket",
    "source_object_key",
    "destination_bucket",
    "destination_object_key",
)

# without checksum verified

clerk.move_object(
    "source_bucket",
    "source_object_key",
    "destination_bucket",
    "destination_object_key",
    verify_checksum=False
)

List Object

clerk.list_objects(
    "bucket_name",
    1000
)

Test

git clone git@bitbucket.org:swiant/object_store_wrapper.git

pip install -e .

export HOST=<host>

export PORT=<port>

export ACCESS_KEY=<access_key>

export SECRET_KEY=<secret_key>

pytest -v object_clerk

Project details

These details have not been verified by PyPI

Project links

Homepage

Release history Release notifications | RSS feed

0.1.1

Feb 16, 2024

0.1.0

Jan 28, 2022

0.1.0rc2 pre-release

Jan 25, 2022

0.0.18

Dec 1, 2021

0.0.18rc1 pre-release

Dec 1, 2021

0.0.17 yanked

Nov 23, 2021

Reason this release was yanked:

Functionality (ArchiveClerk) not supported

This version

0.0.16

Aug 11, 2021

0.0.15 yanked

Aug 11, 2021

Reason this release was yanked:

temporary api change that won't be supported. Use 0.0.16 or later

0.0.14 yanked

Aug 11, 2021

Reason this release was yanked:

temporary api change that won't be supported. Use 0.0.16 or later

0.0.13

Aug 11, 2021

0.0.12

Oct 8, 2020

0.0.11

Oct 7, 2020

0.0.11rc1 pre-release

Oct 7, 2020

0.0.9

Oct 7, 2020

0.0.9rc1 pre-release

Oct 5, 2020

0.0.8

Mar 18, 2020

0.0.7

Feb 24, 2020

0.0.6

Feb 20, 2020

0.0.5

Feb 20, 2020

0.0.4

Feb 18, 2020

0.0.3

Feb 12, 2020

0.0.2

Feb 11, 2020

0.0.1

Feb 7, 2020

0.0.1rc4 pre-release

Feb 7, 2020

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

object-clerk-0.0.16.tar.gz (15.2 kB view details)

Uploaded Aug 11, 2021 Source

File details

Details for the file object-clerk-0.0.16.tar.gz.

File metadata

Download URL: object-clerk-0.0.16.tar.gz
Upload date: Aug 11, 2021
Size: 15.2 kB
Tags: Source
Uploaded using Trusted Publishing? No
Uploaded via: twine/3.4.2 importlib_metadata/4.6.3 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.0 CPython/3.8.11

File hashes

Hashes for object-clerk-0.0.16.tar.gz
Algorithm	Hash digest
SHA256	`9e9317eea20470934770178258761b5846c04f5c2ba63c92998aa841a89569ba`
MD5	`dcbbb11e82e8d9f253c1a7cd910ca7d9`
BLAKE2b-256	`b243266dbae4dd0883fef08ca04689159660666493d773fd87db736ab5c29ba0`