No project description provided
Project description
Exporters provide a flexible way to export data from multiple sources to multiple destinations, allowing filtering and transforming the data.
This Github repository is used as a central repository.
Full documentation can be found here http://exporters.readthedocs.io/en/latest/
Getting Started
Install exporters
First of all, we recommend to create a virtualenv:
virtualenv exporters source exporters/bin/activate
Installing:
pip install exporters
Creating a configuration
- Then, we can create our first configuration object and store it in a file called config.json.
This configuration will read from an s3 bucket and store it in our filesystem, exporting only the records which have United States in field country:
{
"reader": {
"name": "exporters.readers.s3_reader.S3Reader",
"options": {
"bucket": "YOUR_BUCKET",
"aws_access_key_id": "YOUR_ACCESS_KEY",
"aws_secret_access_key": "YOUR_SECRET_KEY",
"prefix": "exporters-tutorial/sample-dataset"
}
},
"filter": {
"name": "exporters.filters.key_value_regex_filter.KeyValueRegexFilter",
"options": {
"keys": [
{"name": "country", "value": "United States"}
]
}
},
"writer":{
"name": "exporters.writers.fs_writer.FSWriter",
"options": {
"filebase": "/tmp/output/"
}
}
}
Export with script
We can use the provided script to run this export:
python bin/export.py --config config.json
Use it as a library
The export can be run using exporters as a library:
from exporters import BasicExporter
exporter = BasicExporter.from_file_configuration('config.json')
exporter.export()
Resuming an export job
Let’s suppose we have a pickle file with a previously failed export job. If we want to resume it we must run the export script:
python bin/export.py --resume pickle://pickle-file.pickle
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for exporters-0.4.14-py2-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bb8dc7144a045a12a2e9f9424b750e1f7cfd82bd8c1dff26e044a0fc1da1f6c2 |
|
MD5 | 607898dc48e5646d9662d3bc6acb3186 |
|
BLAKE2b-256 | a3bbdf931c00ba6353e4d7abbf373c5a0872b93486886208eaa6a06177531f6e |