datapkg 0.7

Obtaining a Package

Search for a package in an Index e.g. on CKAN.net:

# let's search for iso country/language codes data (iso 3166 ...)
$ datapkg search ckan:// iso
...
iso-3166-2-data -- Linked ISO 3166-2 Data
...

Get some information about one of them:

$ datapkg info ckan://iso-3166-2-data

Let’s download it (to the current directory):

$ datapkg download ckan://iso-3166-2-data .

This will download the Package ‘iso-3166-2-data’ together with its “Resources” and unpack it into a directory named ‘iso-3166-2-data’.

Note: we specify packages using ‘package specs’ like ‘ckan://{name}. For more on package ‘specs’ as they are called see the dedicated section below.

Note that, if you replace the ckan:// spec with a file:// spec, you can use most of the commands for files on disk. For example, if you’ve downloaded a data package to, say, /tmp/xyz you could do:

$ datapkg info file:///tmp/xyz

See the help on indivdual commands for more information.

Creating and Registering a Package

Create a new data Package on disk using datapkg file layout:

$ datapkg create MyNewDataPackage

Edit the Package’s metadata:

$ vim MyNewDataPackage/setup.py

Add some data to the Package:

$ cp mydata.csv MyNewDataPackage
$ cp mydata.js MyNewDataPackage
$ etc ...

Register it on CKAN:

# NB: to register on CKAN you'll need to have an api-key
# This can either be stored in your config file (see datapkg init config)
# Or you can set it with the --api-key option
$ datapkg register file://MyNewDataPackage ckan://

Check it has registered ok:

$ datapkg info ckan://mynewdatapkg

You can also upload associated package resources to a storage system. For example, to register {your-file} with the default storage system associated to CKAN in a bucket named after {yourpackagename}:

# this requires you have created your default datapkgrc config file
$ datapkg upload {your-file} ckan://{yourpackagename}/{filename}

Basic Concepts

Before we begin it is useful to understand some basic datapkg concepts:

1. A Package – the ‘package’ of data.

2. A Distribution – a serialization of the Package and optionally the data too (code, database, a book etc) to some concretely addressable form. For example: file(s) on disk, an API at a specific url.

For managing Packages datapkg uses:

1. A Registry: a list of Package metadata (but not Package payload data)

2. Repository: a Registry plus associated storage for association Distributions.

Package Specs (Specifications)

To specify a package (or just an index/repository) we often use a ‘package spec’ (often termed just ‘spec’), which have the following url-like form:

# for CKAN
ckan://{package-name}
# on disk
file://{package-or-index-path}

The use of a ‘naked’ spec, i.e. one without any scheme such as ckan:// or file://, like ‘mypackage’ is used for referring to packages with regard to the default package index.

1. Obtaining Material

$ datapkg init config

2. Making Your Material Available to Others

$ datapkg create {pkg-name-or-path}

$ cd {my-new-package}
$ cp {lots-of-my-data-files} .

$ datapkg register file://{path} ckan://

$ datapkg download {package-spec} {path-on-disk}

3. For Developers

The easiest thing (which also guarantees up-to-date-ness) is to look through the unit tests in ./datapkg/tests/