ckanext-downloadall 0.1.0

Requirements

Designed to work with CKAN 2.7+

Ideally it is used in conjunction with DataStore and xloader (or datapusher), so that the Data Dictionary is included as a schema in the datapackage.json, to describe the column types.

Installation

To install ckanext-downloadall:

1. Activate your CKAN virtual environment, for example:

   . /usr/lib/ckan/default/bin/activate

2. Install the ckanext-downloadall Python package into your virtual environment:

   pip install ckanext-downloadall

3. Add downloadall to the ckan.plugins setting in your CKAN config file (by default the config file is located at /etc/ckan/default/production.ini). e.g.

   ckan.plugins = downloadall

4. Restart the CKAN worker. For example if you’ve deployed it with supervisord:

   sudo supervisorctl restart ckan-worker:ckan-worker-00

5. Restart CKAN server. For example if you’ve deployed CKAN with Apache on Ubuntu:

   sudo service apache2 reload

6. Ensure the background job ‘worker’ process is running - see https://docs.ckan.org/en/2.8/maintaining/background-tasks.html#running-background-jobs

Config Settings

# Include additional fields from the dataset in the datapackage.json (e.g.
# those defined in a ckanext-scheming schema)
# (optional, space separated list).
ckanext.downloadall.dataset_fields_to_add_to_datapackage = district county

Command-line interface

There is a command-line interface:

downloadall --help

Examples of use:

downloadall update-zip gold-prices
downloadall update-all-zips

Troubleshooting

“All resource data” appears as a normal resource, instead of seeing a “Download All” button

You need to enable this extension in the CKAN config and restart the server. See the Installation section above.

ImportError: No module named datapackage

This means you have an older version of ckanapi, which is a dependency of ckanext-downloadall. Install a newer version.

OSError: [Errno 13] Permission denied: ‘/data/ckan/resources/c89’

You are trying to update zips from the command-line but running the tasks synchronously, rather than with the normal worker process. In this case you need to run it as the www-data user e.g.:

sudo -u www-data /usr/lib/ckan/default/bin/downloadall -c /etc/ckan/default/production.ini update-all-zips --synchronous

Development Installation

To install ckanext-downloadall for development, activate your CKAN virtualenv and do:

git clone https://github.com/davidread/ckanext-downloadall.git
cd ckanext-downloadall
python setup.py develop
pip install -r dev-requirements.txt

Remember to run the worker (in a separate terminal):

paster --plugin=ckan jobs worker --config=/etc/ckan/default/development.ini

Running the Tests

To run the tests, do:

nosetests --nologcapture --with-pylons=test.ini

To run the tests and produce a coverage report, first make sure you have coverage installed in your virtualenv (pip install coverage) then run:

nosetests --nologcapture --with-pylons=test.ini --with-coverage --cover-package=ckanext.downloadall --cover-inclusive --cover-erase --cover-tests

Releasing a New Version of ckanext-downloadall

ckanext-downloadall is availabe on PyPI as https://pypi-hypernode.com/project/ckanext-downloadall/. To publish a new version to PyPI follow these steps:

1. Update the version number in the setup.py file. See PEP 440 for how to choose version numbers.

2. Update the CHANGELOG.md with details of this release.

3. Make sure you have the latest version of necessary packages:

   pip install --upgrade setuptools wheel twine

4. Create a source and binary distributions of the new version:

   python setup.py sdist bdist_wheel && twine check dist/*

   Fix any errors you get.

5. Upload the source distribution to PyPI:

   twine upload dist/*

6. Commit any outstanding changes:

   git commit -a
   git push

7. Tag the new release of the project on GitHub with the version number from the setup.py file. For example if the version number in setup.py is 0.0.1 then do:

   git tag 0.0.1
   git push --tags