Skip to main content

OFS - provides plugin-orientated low-level blobstore.

Project description

OFS is a bucket/object storage library.

It provides a common API for storing bitstreams (plus related metadata) in ‘bucket/object’ stores such as:

  • S3, Google Storage, Eucalytus, Archive.org

  • Filesystem (via pairtree)

  • ‘REST’ Store (see remote/reststore.py - implementation at http://bitbucket.org/pudo/repod/)

  • Riak (buggy)

  • add a backend here - just implement the methods in base.py

Why use the library:

  • Abstraction: write common code but use different storage backends

  • More than a filesystem, less than a database - support for metadata as well as bitstreams

Requirements

For all boto-based stores (Google Storage, S3 etc) require boto>=2.0.

Example Usage

(local version - depends on ‘pairtree’, and ‘simplejson’):

 >>> from ofs.local import PTOFS

 >>> o = PTOFS()
 (Equivalent to 'o = PTOFS(storage_dir = "data", uri_base="urn:uuid:", hashing_type="md5")')

 # Claim a bucket - this will add the bucket to the list of existing ones
 >>> uuid_id = o.claim_bucket()
 >>> uuid_id
 '4aaa43cdf5ba44e2ad25acdbd1cf2f70'

 # Choose a bucket name - if it exists, a new UUID one will be formed instead and returned
 >>> bucket_id = o.claim_bucket("foo")
 >>> bucket_id
 'foo'
 >>> bucket_id = o.claim_bucket("foo")
 >>> bucket_id
 '1bf93208521545879e79c13614cd12f0'

 # Store a file:
 >>> o.put_stream(bucket_id, "foo.txt", open("foo....))
 {'_label': 'foo.txt', '_content_length': 10, '_checksum': 'md5:10feda25f8da2e2ebfbe646eea351224', '_last_modified': '2010-08-02T11:37:21', '_creation_date': '2010-08-02T11:37:21'}

 # or:
 >>> o.put_stream(bucket_id, "foo.txt", "asidaisdiasjdiajsidjasidji")
 {'_label': 'foo.txt', '_content_length': 10, '_checksum': 'md5:10feda25f8da2e2ebfbe646eea351224', '_last_modified': '2010-08-02T11:37:21', '_creation_date': '2010-08-02T11:37:21'}

 # adding a file with some parameters:
 >>> o.put_stream(bucket_id, "foooo", "asidaisdiasjdiajsidjasidji", params={"original_uri":"http://...."})
 {'_label': 'foooo', 'original_uri': 'http://....', '_last_modified': '2010-08-02T11:39:11', '_checksum': 'md5:3d690d7e0f4479c5a7038b8a4572d0fe', '_creation_date': '2010-08-02T11:39:11', '_content_length': 26}

 # Get the underlying URL pointing to a resource
 >>> o.get_url(bucket_id, "foo")
   [typical local pairtree response:]
"file:///opt/ofs_store/pairtree_root/1b/f9/32/......./obj/foo"
   [typical remote response]
"http://..."
"ftp://..."

 # adding to existing metadata:
 >>> o.update_metadata(bucket_id, "foooo", {'foo':'bar'})
 {'_label': 'foooo', 'original_uri': 'http://....', '_last_modified': '2010-08-02T11:39:11', '_checksum': 'md5:3d690d7e0f4479c5a7038b8a4572d0fe', '_creation_date': '2010-08-02T11:39:11', '_content_length': 26, 'foo': 'bar'}

 # Remove keys
 >>> o.remove_metadata_keys(bucket_id, "foooo", ['foo'])
 {'_label': 'foooo', 'original_uri': 'http://....', '_last_modified': '2010-08-02T11:39:11', '_checksum': 'md5:3d690d7e0f4479c5a7038b8a4572d0fe', '_creation_date': '2010-08-02T11:39:11', '_content_length': 26}

 # Delete blob
 >>> o.exists(bucket_id, "foooo")
 True
 >>> o.del_stream(bucket_id, "foooo")
 >>> o.exists(bucket_id, "foooo")
 False

 # Iterate through ids for buckets held:
 >>> for item in o.list_buckets():
 ...   print(item)
 ...
 447536aa0f1b411089d12399738ede8e
 4a726b0a33974480a2a26d34fa0d494d
 4aaa43cdf5ba44e2ad25acdbd1cf2f70
 .... etc

 # Display the labels in a specific bucket:
 >>>o.list_labels("1bf93208521545879e79c13614cd12f0")
 [u'foo.txt']

Developer

Tests use plain unittest but recommend using nose.

To run the botostore tests you’ll need to copy test.ini.tmpl to test.ini and put in details for a google storage account.

Changelog

v0.4.1: 2011-08-13

  • Set checksum (md5) based on etag (botostore backends) if not set

v0.4: 2011-04-28

  • New authenticate_request method for boto based backends.

  • Improved update_medata in botostore (no need to download and re-upload).

v0.3: 2011-01-20

  • S3Bounce backend (use authorization credentials from CKAN).

  • Use setuptools plugins with ofs.backend to allow for 3rd party backends

  • ofs_upload command

v0.2: 2010-11-20

  • Google Storage support.

  • REST store

v0.1: 2010-10-14

  • Initial implemenation with PairTree and S3

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

ofs-0.4.3.tar.gz (153.6 kB view details)

Uploaded Source

File details

Details for the file ofs-0.4.3.tar.gz.

File metadata

  • Download URL: ofs-0.4.3.tar.gz
  • Upload date:
  • Size: 153.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for ofs-0.4.3.tar.gz
Algorithm Hash digest
SHA256 aa57c237a0b3af44e988a4dbc8d504610c86851303338708eb39ff32766281f0
MD5 3553691dea6a902a400c6c4e9b71264d
BLAKE2b-256 5990d7b380d3c7098a3e6df8b95c679bd8a89bd92d6617d7740c0f03faaf4c85

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page