Skip to main content

A library for mapping CKAN metadata <=> Frictionless metadata.

Project description

Frictionless CKAN Mapper

A library for mapping CKAN metadata <=> Frictionless metadata.

The library has zero dependencies (not even on Data Package libs). You can use it directly or use it for inspiration. Detailed outline of the algorithm is in the docs or you can read the code.

Travis Coveralls PyPi SemVer Chat on Discord

Installation

  • Python: install Python. The library is compatible with both Python 2.7+ and Python 3.3+.
pip install frictionless-ckan-mapper

Note: The package is installed as frictionless-ckan-mapper and then imported as frictionless_ckan_mapper.

Getting started

CKAN => Frictionless

# get a CKAN metadata item
ckan_dataset = {
  "name": "my-dataset",
  "title": "My awesome dataset",
  "url": "http://www.example.com/data.csv"
}

# or load from an API e.g.
# ckan_dataset = json.load(urllib.urlopen(
#     https://demo.ckan.org/api/3/package_show?id=my_dataset
# ))

from frictionless_ckan_mapper import ckan_to_frictionless as converter

# convert to frictionless
frictionless_package = converter.dataset(ckan_dict)

print(frictionless_package)

Frictionless => CKAN

frictionless = {
  'name': "f11s-dataset",
  'path': "https://datahub.io/data.csv"
}

from frictionless_ckan_mapper import frictionless_to_ckan as f2c

ckanout = f2c.dataset(frictionless)

print(ckanout)

Reference

This package contains two modules:

  • frictionless_to_ckan
  • ckan_to_frictionless

You can import them directly like so:

from frictionless_ckan_mapper import ckan_to_frictionless
from frictionless_ckan_mapper import frictionless_to_ckan

ckan_to_frictionless

resource(ckandict)

from frictionless_ckan_mapper import ckan_to_frictionless as converter

# ... Some code with a CKAN dictionary ...

output_frictionless_dict = converter.resource(ckan_dictionary)

dataset(ckandict)

from frictionless_ckan_mapper import ckan_to_frictionless as converter

# ... Some code with a CKAN dictionary ...

output_frictionless_dict = converter.dataset(ckan_dictionary)

frictionless_to_ckan

resource(fddict)

from frictionless_ckan_mapper import frictionless_to_ckan as converter

# ... Some code with a Frictionless dictionary ...

output_ckan_dict = converter.resource(frictionless_dictionary)

package(fddict)

from frictionless_ckan_mapper import frictionless_to_ckan as converter

# ... Some code with a Frictionless dictionary ...

output_ckan_dict = converter.package(frictionless_dictionary)

Design

Frictionless   <=>        CKAN
--------------------------------------
Data Package   <=>   Package (Dataset)
Data Resource  <=>   Resource
Table Schema   <=>   Data Dictionary?? (datastore resources can have schemas)

CKAN reference

Summary:

classDiagram

class Package
class Resource
class DataDictionary

Package *-- Resource
Resource o-- DataDictionary

mermaid-diagram-20200703112520

Source for CKAN metadata structure:

Algorithm: CKAN => Frictionless

See the code in frictionless_ckan_mapper/ckan_to_frictionless.py

Algorithm: Frictionless => CKAN

See the code in frictionless_ckan_mapper/frictionless_to_ckan.py

Developers

Install the source

  • Clone the repo:

    git clone https://github.com/frictionlessdata/frictionless-ckan-mapper.git
    
  • And install it with pip:

    pip install -e .
    

Run the tests

Use the excellent pytest suite as follows:

pytest tests

To test under both Python 2 and Python 3 environments, we use tox. You can run the following command:

make test

Note: Make sure that the necessary Python versions are in your environment PATH (Python 2.7 and Python 3.6).

Building and publishing the package

To see a list of available commands from the Makefile, execute:

make list

Build the distribution package locally for testing purposes

If a previous build exists, make sure to also remove it before building again:

make distclean

Then:

make dist

Alternatively, this command will accomplish the same to build packages for both Python 2 and Python 3:

python setup.py sdist bdist_wheel --universal

Test the package at test.pypi.org

python -m twine upload --repository testpypi dist/*

The package will be publicly available at https://test.pypi.org/project/frictionless-ckan-mapper/ and you will be able to pip install it as usual.

Tag a new Git release and publish to the official PyPi

Make sure to update the version of the package in the file frictionless_ckan_mapper/VERSION. Then:

make release

You can quickly review the version to release with make version, which will print the current version stored in VERSION.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

frictionless-ckan-mapper-1.0.9.tar.gz (16.2 kB view details)

Uploaded Source

Built Distribution

frictionless_ckan_mapper-1.0.9-py2.py3-none-any.whl (9.7 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file frictionless-ckan-mapper-1.0.9.tar.gz.

File metadata

File hashes

Hashes for frictionless-ckan-mapper-1.0.9.tar.gz
Algorithm Hash digest
SHA256 9972b94f59709bf83928a02f9581a42981364a69c2a150053178427432b37893
MD5 b4d3151fc3c85ea4f493f8b822183f04
BLAKE2b-256 a4b507ba152e84f3b48dc3a8542dd5d8a8bb2212199809a338d1ed430e20bbdd

See more details on using hashes here.

File details

Details for the file frictionless_ckan_mapper-1.0.9-py2.py3-none-any.whl.

File metadata

File hashes

Hashes for frictionless_ckan_mapper-1.0.9-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 d95af2d5a56bf1a300a0a20c713067e98d0b71a54cd7dfc3acb737dd4db3f968
MD5 c09b033c4d0ace02a40d6a397df856e7
BLAKE2b-256 b9602e01c7e6fca00f3763f24ea1f960a0830709eba36846f013740341e783d9

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page