{{ DESCRIPTION }}
Project description
Data Package Pipelines processors for CKAN.
Install
# clone the repo and install it with pip git clone https://github.com/frictionlessdata/datapackage-pipelines-ckan.git pip install -e .
Usage
datapackage-pipelines-ckan contains several pipeline processors for working with CKAN.
ckan.add_ckan_resource
A processor to retrieve metadata about a CKAN resource from a CKAN instance and add it as a datapackage resource.
run: ckan.add_ckan_resource
parameters:
ckan-host: http://demo.ckan.org
resource-id: d51c9bd4-8256-4289-bdd7-962f8572efb0
ckan-api-key: env:CKAN_API_KEY # an env var defining a ckan user api keyckan-host: The base url (and scheme) for the CKAN instance (e.g. http://demo.ckan.org).
resource-id: The id of CKAN resource
ckan-api-key: Either a CKAN user api key or, if in the format env:CKAN_API_KEY_NAME, an env var that defines an api key. Optional, but necessary for private datasets.
ckan.dump.to_ckan
A processor to save a datapackage and resources to a specified CKAN instance.
run: ckan.dump.to_ckan
parameters:
ckan-host: http://demo.ckan.org
ckan-api-key: env:CKAN_API_KEY
overwrite_existing: true
push_resources_to_datastore: true
dataset-properties:
name: test-dataset-010203
state: draft
private: true
owner_org: my-test-orgckan-host: The base url (and scheme) for the CKAN instance (e.g. http://demo.ckan.org).
ckan-api-key: Either a CKAN user api key or, if in the format env:CKAN_API_KEY_NAME, an env var that defines an api key.
overwrite_existing: If true, if the CKAN dataset already exists, it will be overwritten by the datapackage. Optional, and default is false.
push_resources_to_datastore: If true, newly created resources will be pushed the CKAN DataStore. Optional, and default is false.
push_resources_to_datastore_method: Value is a string, one of ‘upsert’, ‘insert’ or ‘update’. This will be the method used to add data to the DataStore (see https://ckan.readthedocs.io/en/latest/maintaining/datastore.html#ckanext.datastore.logic.action.datastore_upsert). Optional, the default is ‘insert’.
dataset-properties: An optional object, the properties of which will be used to set properties of the CKAN dataset.
CKAN dataset from datapackage
The processor first creates a CKAN dataset from the datapackage specification, using the CKAN api `package_create <http://docs.ckan.org/en/latest/api/#ckan.logic.action.create.package_create>`__. If the dataset already exists, and parameter overwrite_existing is True, the processor will attempt to update the CKAN dataset using `package_update <http://docs.ckan.org/en/latest/api/#ckan.logic.action.update.package_update>`__. All existing resources and dataset properties will be overwritten.
CKAN resources from datapackage resources
If the CKAN dataset was successfully created or updated, the dataset resources will be created for each resource in the datapackage, using `resource_create <http://docs.ckan.org/en/latest/api/#ckan.logic.action.create.resource_create>`__. If datapackage resource are marked for streaming (they have the dpp:streamed=True property), resource files will be uploaded to the CKAN filestore. For example, remote resources may be marked for streaming by the inclusion of the stream_remote_resources processor earlier in the pipeline.
Additionally, if push_resources_to_datastore is True, the processor will push resources marked for streaming to the CKAN DataStore using `datastore_create <https://ckan.readthedocs.io/en/latest/maintaining/datastore.html#ckanext.datastore.logic.action.datastore_create>`__ and `datastore_upsert <https://ckan.readthedocs.io/en/latest/maintaining/datastore.html#ckanext.datastore.logic.action.datastore_upsert>`__.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file datapackage-pipelines-ckan-0.0.2b0.tar.gz.
File metadata
- Download URL: datapackage-pipelines-ckan-0.0.2b0.tar.gz
- Upload date:
- Size: 93.6 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 1c08f59ac0aa8d04e8610354d37da30f4eaff446ad0d04bff9ff99a209f2a9b9 |
|
| MD5 | 0493dfc42b26dca585ee3ae127ecb069 |
|
| BLAKE2b-256 | 0cfd05f410abde277e6cd95ae258de4f56b0c89f8a41148e3cbd318b6e829141 |
File details
Details for the file datapackage_pipelines_ckan-0.0.2b0-py2.py3-none-any.whl.
File metadata
- Download URL: datapackage_pipelines_ckan-0.0.2b0-py2.py3-none-any.whl
- Upload date:
- Size: 10.8 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | df9946173a2de95ccf67e1f277b9409e5fa032b1128d2ed7bcf3146e92e5c264 |
|
| MD5 | c15249c9cde33748d2a322eb1be92ef7 |
|
| BLAKE2b-256 | 85110eca89aad50d429d05e48e637e734636e05931a027617dd55f2d870d222a |
