High-level Python abstraction layer for Amazon S3
Project description
baiji
=====
High-level Python abstraction layer for Amazon S3:
1. An [`open`][open]-like context handler which allows using S3 keys and
local files interchangeably.
- When reading S3, contents are first written to a temporary local
file.
- When writing S3, contents are written to a temporary local file,
and uploaded on close.
2. An `s3` CLI for listing, copying, syncing, and other common activities.
Features
--------
- Works without an S3 connection (with local files).
- Supports multiprocess parallelism for copying lots of files.
- Supports Python 2.7 and uses boto2.
- Supports OS X, Linux, and Windows.
- Tested and production-hardened.
[open]: https://docs.python.org/2/library/functions.html#open
Examples
--------
```py
with s3.open('s3://example/info.txt', 'w') as f:
f.write('hello')
with s3.open('file:///home/me/info.txt', 'w') as f:
f.write('hello')
with s3.open('s3://example/info.txt', 'r') as f:
contents = f.readlines()
with s3.open('file:///home/me/info.txt', 'r') as f:
contents = f.readlines()
```
```sh
s3 cp foo.txt s3://example/bar.txt
s3 cp s3://example/bar.txt s3://another-example/bazinga.txt
s3 rm s3://example/bar.txt
```
Development
-----------
```sh
pip install -r requirements_dev.txt
rake test
rake lint
```
TODO
----
1. Migrate credentials to `~/.aws/credentials` or env, and deprecate AWS
credential support in `~/.bodylabs`.
2. Move `baiji.util.parallel` into a separate library.
3. Upgrade to boto3.
Contribute
----------
- Issue Tracker: github.com/bodylabs/baiji/issues
- Source Code: github.com/bodylabs/baiji
Pull requests welcome!
Support
-------
If you are having issues, please let us know.
License
-------
The project is licensed under the Apache license, version 2.0.
=====
High-level Python abstraction layer for Amazon S3:
1. An [`open`][open]-like context handler which allows using S3 keys and
local files interchangeably.
- When reading S3, contents are first written to a temporary local
file.
- When writing S3, contents are written to a temporary local file,
and uploaded on close.
2. An `s3` CLI for listing, copying, syncing, and other common activities.
Features
--------
- Works without an S3 connection (with local files).
- Supports multiprocess parallelism for copying lots of files.
- Supports Python 2.7 and uses boto2.
- Supports OS X, Linux, and Windows.
- Tested and production-hardened.
[open]: https://docs.python.org/2/library/functions.html#open
Examples
--------
```py
with s3.open('s3://example/info.txt', 'w') as f:
f.write('hello')
with s3.open('file:///home/me/info.txt', 'w') as f:
f.write('hello')
with s3.open('s3://example/info.txt', 'r') as f:
contents = f.readlines()
with s3.open('file:///home/me/info.txt', 'r') as f:
contents = f.readlines()
```
```sh
s3 cp foo.txt s3://example/bar.txt
s3 cp s3://example/bar.txt s3://another-example/bazinga.txt
s3 rm s3://example/bar.txt
```
Development
-----------
```sh
pip install -r requirements_dev.txt
rake test
rake lint
```
TODO
----
1. Migrate credentials to `~/.aws/credentials` or env, and deprecate AWS
credential support in `~/.bodylabs`.
2. Move `baiji.util.parallel` into a separate library.
3. Upgrade to boto3.
Contribute
----------
- Issue Tracker: github.com/bodylabs/baiji/issues
- Source Code: github.com/bodylabs/baiji
Pull requests welcome!
Support
-------
If you are having issues, please let us know.
License
-------
The project is licensed under the Apache license, version 2.0.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
baiji-2.6.1.tar.gz
(39.0 kB
view details)
Built Distribution
baiji-2.6.1-py2-none-any.whl
(48.8 kB
view details)
File details
Details for the file baiji-2.6.1.tar.gz
.
File metadata
- Download URL: baiji-2.6.1.tar.gz
- Upload date:
- Size: 39.0 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4c902b28140ee688a7383fe5d3e636de9f69972ae832467fa1a3a2e47cf8c8f2 |
|
MD5 | 84397f80e7eb453983a236b9804e3b48 |
|
BLAKE2b-256 | f1469d2c776562ba186fbb304efda079fe0acd4bae7d8af941f19cb1dacc96c9 |
Provenance
File details
Details for the file baiji-2.6.1-py2-none-any.whl
.
File metadata
- Download URL: baiji-2.6.1-py2-none-any.whl
- Upload date:
- Size: 48.8 kB
- Tags: Python 2
- Uploaded using Trusted Publishing? No
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 09f1d911db88463a5ffa7914904aaab12e47a4d987172829270cf02b4cd4e789 |
|
MD5 | 31281c3a4c5447898b19c87fa3f62a05 |
|
BLAKE2b-256 | 670876135659918a010a7aefdf7dc2c642dfbfd65a23d324c0784a5e8706f8db |