ckanext-xloader 1.2.0

Key differences from DataPusher

Requirements

Compatibility with core CKAN versions:

Installation

To install XLoader:

1. Activate your CKAN virtual environment, for example:

   . /usr/lib/ckan/default/bin/activate

2. Install the ckanext-xloader Python package into your virtual environment:

   pip install ckanext-xloader

3. Install dependencies:

   pip install -r https://raw.githubusercontent.com/ckan/ckanext-xloader/master/requirements.txt
   pip install -U requests[security]

4. Add xloader to the ckan.plugins setting in your CKAN config file (by default the config file is located at /etc/ckan/default/production.ini).

   You should also remove datapusher if it is in the list, to avoid them both trying to load resources into the DataStore.

   Ensure datastore is also listed, to enable CKAN DataStore.

5. Starting CKAN 2.10 you will need to set an API Token to be able to execute jobs against the server:

   ckanext.xloader.api_token = <your-CKAN-generated-API-Token>

6. If it is a production server, you’ll want to store jobs info in a more robust database than the default sqlite file. It can happily use the main CKAN postgres db by adding this line to the config, but with the same value as you have for sqlalchemy.url:

   ckanext.xloader.jobs_db.uri = postgresql://ckan_default:pass@localhost/ckan_default

   (This step can be skipped when just developing or testing.)

7. Restart CKAN. For example if you’ve deployed CKAN with Apache on Ubuntu:

   sudo service apache2 reload

8. Run the worker:

   ckan -c /etc/ckan/default/ckan.ini jobs worker

Config settings

Configuration:

See the extension’s config_declaration.yaml file.

This plugin also supports the ckan.download_proxy setting, to use a proxy server when downloading files. This setting is shared with other plugins that download resource files, such as ckanext-archiver. Eg:

ckan.download_proxy = http://my-proxy:1234/

You may also wish to configure the database to use your preferred date input style on COPY. For example, to make [PostgreSQL](https://www.postgresql.org/docs/current/runtime-config-client.html#RUNTIME-CONFIG-CLIENT-FORMAT) expect European (day-first) dates, you could add to postgresql.conf:

datestyle=ISO,DMY

External Database credentials for datastore

ckanext.xloader.jobs_db.uri = postgresql://ckan_default:pass@localhost/ckan_default

API Key requires for xloader interaction CKAN 2.10 onwards, to generate TOKEN=ckan -c /etc/ckan/default/production.ini user token add $ACCOUNT xloader | tail -1 | tr -d '[:space:]')

ckanext.xloader.api_token = <your-CKAN-generated-API-Token>

Badge notification on what xloader is doing

ckanext.xloader.show_badges = True|False (default True)

ckanext.xloader.debug_badges = True|False (default False)

Developer installation

To install XLoader for development, activate your CKAN virtualenv and in the directory up from your local ckan repo:

git clone https://github.com/ckan/ckanext-xloader.git
cd ckanext-xloader
pip install -e .
pip install -r requirements.txt
pip install -r dev-requirements.txt

Upgrading from DataPusher

To upgrade from DataPusher to XLoader:

1. Install XLoader as above, including running the xloader worker.

2. (Optional) For existing datasets that have been datapushed to datastore, freeze the column types (in the data dictionaries), so that XLoader doesn’t change them back to string on next xload:

   ckan -c /etc/ckan/default/ckan.ini migrate_types

3. If you’ve not already, change the enabled plugin in your config - on the ckan.plugins line replace datapusher with xloader.

4. (Optional) If you wish, you can disable the direct loading and continue to just use tabulator - for more about this see the docs on config option: ckanext.xloader.use_type_guessing

5. Stop the datapusher worker:

   sudo a2dissite datapusher

6. Restart CKAN:

   sudo service apache2 reload
   sudo service nginx reload

Command-line interface

You can submit single or multiple resources to be xloaded using the command-line interface.

e.g.

ckan -c /etc/ckan/default/ckan.ini xloader submit <dataset-name>

For debugging you can try xloading it synchronously (which does the load directly, rather than asking the worker to do it) with the -s option:

ckan -c /etc/ckan/default/ckan.ini xloader submit <dataset-name> -s

See the status of jobs:

ckan -c /etc/ckan/default/ckan.ini xloader status

Submit all datasets’ resources to the DataStore:

ckan -c /etc/ckan/default/ckan.ini xloader submit all

Re-submit all the resources already in the DataStore (Ignores any resources that have not been stored in DataStore e.g. because they are not tabular):

ckan -c /etc/ckan/default/ckan.ini xloader submit all-existing

Full list of XLoader CLI commands:

ckan -c /etc/ckan/default/ckan.ini xloader --help

ckan -c /etc/ckan/default/ckan.ini jobs clear [QUEUES]

sudo supervisorctl restart ckan-worker:*

Troubleshooting

KeyError: “Action ‘datastore_search’ not found”

You need to enable the datastore plugin in your CKAN config. See ‘Installation’ section above to do this and restart the worker.

ProgrammingError: (ProgrammingError) relation “_table_metadata” does not exist

Your DataStore permissions have not been set-up - see: <https://docs.ckan.org/en/latest/maintaining/datastore.html#set-permissions>

Running the Tests

The first time, your test datastore database needs the trigger applied:

sudo -u postgres psql datastore_test -f full_text_function.sql

To run the tests, do:

pytest ckan-ini=test.ini ckanext/xloader/tests

Releasing a New Version of XLoader

XLoader is available on PyPI as https://pypi-hypernode.com/project/ckanext-xloader.

To publish a new version to PyPI follow these steps:

1. Update the version number in the setup.py file. See PEP 440 for how to choose version numbers.

2. Update the CHANGELOG.

3. Make sure you have the latest version of necessary packages:

   pip install --upgrade setuptools wheel twine

4. Create source and binary distributions of the new version:

   python setup.py sdist bdist_wheel && twine check dist/*

   Fix any errors you get.

5. Upload the source distribution to PyPI:

   twine upload dist/*

6. Commit any outstanding changes:

   git commit -a
   git push

7. Tag the new release of the project on GitHub with the version number from the setup.py file. For example if the version number in setup.py is 0.0.1 then do:

   git tag 0.0.1
   git push --tags