object-clerk

A wrapper for select object operations using boto3 for Ceph which confirms integrity of reads and heals intermittent connectivity.

These details have not been verified by PyPI

Project links

Homepage

Project description

A wrapper for the following boto3 s3 client operations with connection retry and checksum verification:

get_object
head_object
upload_fileobj
copy_object
delete_object
list_objects_v2

Features

Retry connection failures
Confirm checksum of uploaded and retrieved objects
Move object
Constrained interface to support simple CRUD operations for objects in existing buckets

Installation

pip install object-clerk

Examples

Initialize

clerk = ObjectClerk(host=127.0.0.1, port=8080, access_key=12342, secret_key=12342, retry_delay=1, retry_backoff=1, retry_jitter=(1, 3), retry_max_delay=5, retry_tries=3, use_ssl=False)'

Get Object

# with checksum verified

bytes_response = clerk.get_object("bucket", "object_key")

# without checksum verified

bytes_response = clerk.get_object("bucket", "object_key", verify_checksum=False)

Delete Object

clerk.delete_object("bucket", "object_key")

Get Object Info

dict_response = clerk.get_object_info("bucket", "object_key")

Copy Object

# with checksum verified

clerk.copy_object(
    "source_bucket",
    "source_object_key",
    "destination_bucket",
    "destination_object_key",
)

# without checksum verified

clerk.copy_object(
    "source_bucket",
    "source_object_key",
    "destination_bucket",
    "destination_object_key",
    verify_checksum=False
)

Upload Object

# with checksum verified

with open("file", mode='rb') as f:

    clerk.upload_object(f, "bucket", "object_key")

# without checksum verified

with open("file", mode='rb') as f:

    clerk.upload_object(f, "bucket", "object_key", verify_checksum=False)

Move Object

# with checksum verified

clerk.move_object(
    "source_bucket",
    "source_object_key",
    "destination_bucket",
    "destination_object_key",
)

# without checksum verified

clerk.move_object(
    "source_bucket",
    "source_object_key",
    "destination_bucket",
    "destination_object_key",
    verify_checksum=False
)

List Object

clerk.list_objects(
    "bucket_name",
    1000
)

Test

git clone git@bitbucket.org:swiant/object_store_wrapper.git

pip install -e .

export HOST=<host>

export PORT=<port>

export ACCESS_KEY=<access_key>

export SECRET_KEY=<secret_key>

pytest -v object_clerk

Project details

These details have not been verified by PyPI

Project links

Homepage

Release history Release notifications | RSS feed

0.1.1

Feb 16, 2024

0.1.0

Jan 28, 2022

0.1.0rc2 pre-release

Jan 25, 2022

0.0.18

Dec 1, 2021

0.0.18rc1 pre-release

Dec 1, 2021

0.0.17 yanked

Nov 23, 2021

Reason this release was yanked:

Functionality (ArchiveClerk) not supported

0.0.16

Aug 11, 2021

0.0.15 yanked

Aug 11, 2021

Reason this release was yanked:

temporary api change that won't be supported. Use 0.0.16 or later

0.0.14 yanked

Aug 11, 2021

Reason this release was yanked:

temporary api change that won't be supported. Use 0.0.16 or later

0.0.13

Aug 11, 2021

0.0.12

Oct 8, 2020

0.0.11

Oct 7, 2020

0.0.11rc1 pre-release

Oct 7, 2020

0.0.9

Oct 7, 2020

This version

0.0.9rc1 pre-release

Oct 5, 2020

0.0.8

Mar 18, 2020

0.0.7

Feb 24, 2020

0.0.6

Feb 20, 2020

0.0.5

Feb 20, 2020

0.0.4

Feb 18, 2020

0.0.3

Feb 12, 2020

0.0.2

Feb 11, 2020

0.0.1

Feb 7, 2020

0.0.1rc4 pre-release

Feb 7, 2020

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

object-clerk-0.0.9rc1.tar.gz (14.8 kB view details)

Uploaded Oct 5, 2020 Source

File details

Details for the file object-clerk-0.0.9rc1.tar.gz.

File metadata

Download URL: object-clerk-0.0.9rc1.tar.gz
Upload date: Oct 5, 2020
Size: 14.8 kB
Tags: Source
Uploaded using Trusted Publishing? No
Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/50.3.0 requests-toolbelt/0.9.1 tqdm/4.50.0 CPython/3.8.6

File hashes

Hashes for object-clerk-0.0.9rc1.tar.gz
Algorithm	Hash digest
SHA256	`2564b88ca74c027f7664425c2e7bd0e2235d9c04c5b7c0ab072ef5e197d301a9`
MD5	`9e4d83b28aa46297154201302f621960`
BLAKE2b-256	`8278bd0efc0d1901bf86daa3d8f192c9916a3589c0911602a5693c6459af5e8f`