Generate CKAN DataStore tables, load and extract data, based on Table Schema descriptors.
Project description
tableschema-ckan-datastore-py
Generate and load CKAN DataStore tables based on Table Schema descriptors.
Features
- implements
tableschema.Storage
interface
Contents
Getting Started
Installation
The package use semantic versioning. It means that major versions could include breaking changes. It's highly recommended to specify package
version range in your setup/requirements
file e.g. package>=1.0,<2.0
.
pip install tableschema-ckan-datastore
Documentation
When writing data, tableschema-ckan-datastore uses the CKAN API datastore_upsert
endpoint with the upsert
method. This requires a unique key in the data defined by a Table Schema primary key property. If your data has a primary key, you can use the table.save
method:
from tableschema import Table
# Load and save CKAN DataStore record
resource_id = 'bd79c992-40f0-454a-a0ff-887f84a792fb'
base_url = 'https://demo.ckan.org'
dataset_id = 'test-dataset-010203'
api_key = 'my-ckan-user-api-key'
table = Table('data.csv', schema='schema.json') # data.csv has primary keys
table.save(resource_id,
storage='ckan_datastore',
base_url=base_url,
dataset_id=dataset_id,
api_key=api_key)
If you need to define the method used to save data to the DataStore, you can create the tableschema.Storage
object directly and specify which method parameter to use:
import io
import json
from tabulator import Stream
from tableschema import Storage
# Load and save CKAN DataStore record
resource_id = 'bd79c992-40f0-454a-a0ff-887f84a792fb'
base_url = 'https://demo.ckan.org'
dataset_id = 'test-dataset-010203'
api_key = 'my-ckan-user-api-key'
schema = json.load(io.open('schema.json', encoding='utf-8'))
data = Stream('data.csv', headers=1).open()
storage = Storage.connect('ckan_datastore',
base_url=base_url,
dataset_id=dataset_id,
api_key=api_key)
storage.create(resource_id, schema, force=True))
storage.write(resource_id, data, method='insert') # specify the datastore_upsert method
API Reference
Storage
Storage(self, base_url, dataset_id=None, api_key=None)
Ckan Datastore storage
Package implements Tabular Storage interface (see full documentation on the link):
Only additional API is documented
Arguments
- base_url (str):
- the base url (and scheme) for the CKAN instance (e.g. http://demo.ckan.org).
- dataset_id (str): id or name of the CKAN dataset we wish to use as the bucket source. If missing, all tables in the DataStore are used.
- api_key (str):
either a CKAN user api key or, if in the format
env:CKAN_API_KEY_NAME
, an env var that defines an api key.
Contributing
The project follows the Open Knowledge International coding standards.
Recommended way to get started is to create and activate a project virtual environment. To install package and development dependencies into active environment:
$ make install
To run tests with linting and coverage:
$ make test
Changelog
Here described only breaking and the most important changes. The full changelog and documentation for all released versions could be found in nicely formatted commit history.
v1.1
- Increase max size of a bucket to 100 resources
v1.0
- Initial driver implementation
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for tableschema-ckan-datastore-1.1.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | b82e1f8de6d72f927e8b7b146d69c8c7ccb57105a4c07899e5b81e0371beabc7 |
|
MD5 | ffba67acc6c26064a2cf6ffd693b4fda |
|
BLAKE2b-256 | 12716e478c1326ab4d4b1b6c3ec425e40bdc9741d5098c0152fbdf2982ca6a2f |
Hashes for tableschema_ckan_datastore-1.1.1-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cdf8c3e13984e045a583942f379bf20be06b8d5acf8268b5853a258b1304ee4c |
|
MD5 | 5fd3863b30471af7e0bff2c9b93c52be |
|
BLAKE2b-256 | 2f1c7e1ca5ad1442ef1314652e61f63ed6357515f32e34a89c4c042187676a30 |