Skip to main content

Generate CKAN DataStore tables, load and extract data, based on Table Schema descriptors.

Project description

tableschema-ckan-datastore-py

Travis Coveralls PyPi Github Gitter

Generate and load CKAN DataStore tables based on Table Schema descriptors.

Features

  • implements tableschema.Storage interface

Contents

Getting Started

Installation

The package use semantic versioning. It means that major versions could include breaking changes. It's highly recommended to specify package version range in your setup/requirements file e.g. package>=1.0,<2.0.

pip install tableschema-ckan-datastore

Documentation

When writing data, tableschema-ckan-datastore uses the CKAN API datastore_upsert endpoint with the upsert method. This requires a unique key in the data defined by a Table Schema primary key property. If your data has a primary key, you can use the table.save method:

from tableschema import Table

# Load and save CKAN DataStore record
resource_id = 'bd79c992-40f0-454a-a0ff-887f84a792fb'
base_url = 'https://demo.ckan.org'
dataset_id = 'test-dataset-010203'
api_key = 'my-ckan-user-api-key'

table = Table('data.csv', schema='schema.json')  # data.csv has primary keys
table.save(resource_id,
           storage='ckan_datastore',
           base_url=base_url,
           dataset_id=dataset_id,
           api_key=api_key)

If you need to define the method used to save data to the DataStore, you can create the tableschema.Storage object directly and specify which method parameter to use:

import io
import json
from tabulator import Stream
from tableschema import Storage

# Load and save CKAN DataStore record
resource_id = 'bd79c992-40f0-454a-a0ff-887f84a792fb'
base_url = 'https://demo.ckan.org'
dataset_id = 'test-dataset-010203'
api_key = 'my-ckan-user-api-key'

schema = json.load(io.open('schema.json', encoding='utf-8'))
data = Stream('data.csv', headers=1).open()

storage = Storage.connect('ckan_datastore',
                          base_url=base_url,
                          dataset_id=dataset_id,
                          api_key=api_key)
storage.create(resource_id, schema, force=True))
storage.write(resource_id, data, method='insert')  # specify the datastore_upsert method

API Reference

Storage

Storage(self, base_url, dataset_id=None, api_key=None)

Ckan Datastore storage

Package implements Tabular Storage interface (see full documentation on the link):

Storage

Only additional API is documented

Arguments

  • base_url (str):
  • the base url (and scheme) for the CKAN instance (e.g. http://demo.ckan.org).
  • dataset_id (str): id or name of the CKAN dataset we wish to use as the bucket source. If missing, all tables in the DataStore are used.
  • api_key (str): either a CKAN user api key or, if in the format env:CKAN_API_KEY_NAME, an env var that defines an api key.

Contributing

The project follows the Open Knowledge International coding standards.

Recommended way to get started is to create and activate a project virtual environment. To install package and development dependencies into active environment:

$ make install

To run tests with linting and coverage:

$ make test

Changelog

Here described only breaking and the most important changes. The full changelog and documentation for all released versions could be found in nicely formatted commit history.

v1.1

  • Increase max size of a bucket to 100 resources

v1.0

  • Initial driver implementation

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tableschema-ckan-datastore-1.1.1.tar.gz (10.8 kB view hashes)

Uploaded Source

Built Distribution

tableschema_ckan_datastore-1.1.1-py2.py3-none-any.whl (9.8 kB view hashes)

Uploaded Python 2 Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page