mr.migrator 1.2

Introduction

Transmogrifier is a powerful tool for creating transformation processes called “pipelines”. Transmogrifier gives you the tools to create and share these pipelines but doesn’t provide an easy way to run the pipeline. Mr.migrator aims to bridge that gap.

Mr.Migrator provides the following:

• A buildout recipe with which you can override a given pipeline.

• A commandline script (with help) which lets you run pipelines and see their progress. This is useful when used in conjunction with transmogrify.ploneremote or other blueprints which are not expected to be run inside the Plone process.

• A Plone plugin that once installed lets you pick which pipeline you want to run, provides a form to override the pipeline options and then provides progress on the running pipeline. This is useful when you want to use plone.app.transmogrifier blueprints that should be run inside the Plone process.

Getting a pipeline

A pipeline is a concept from collective.transmogrifier where dictionary items pass through a series of steps, each adding, removing or uploading information to an external source. A pipeline is configured in configuration file using the INI style syntax. Mr.Migrator lets you run either pipelines you built yourself, or

migrate --pipeline=mypipeline.cfg

[migrate]
recipe = mr.migrator
pipeline = mypipeline.cfg

[buildout]
parts += mr.migrator

[migrator]
recipe = mr.migrator
pipeline = mypipeline.cfg
eggs = transmogrify.sqlalchemy
zcml = transmogrify.sqlalchemy

migrate --zcml=transmogrify.sqlalchemy,transmogrify.other

entry-points = {"z3c.autoinclude.plugin":['target = transmogrify']}

<configure
    xmlns="http://namespaces.zope.org/zope"
    xmlns:transmogrifier="http://namespaces.plone.org/transmogrifier"
    i18n_domain="collective.transmogrifier">

<transmogrifier:registerConfig
    name="exampleconfig"
    title="Example pipeline configuration"
    description="This is an example pipeline configuration"
    configuration="example.cfg"
    />

</configure>

entry_points = {"z3c.autoinclude.plugin":['target = transmogrify']}

migrate --pipeline=exampleconfig

[migrate]
recipe = mr.migrator
pipeline = exampleconfig

Overriding pipeline values

Pipelines are organised as a series of steps through which crawled items pass before eventually being uploaded. Each step as one or more configuration options so you can customise import process for your needs. Almost all imports will require some level of configurations.

The first part of each configuration key is the step e.g. crawler. The second part is the particular configuration option for that particular step. e.g. url. This is then followed by = and value or values.

The configuration options can either be given as part of the buildout part e.g.

[buildout]
parts += mr.migrator

[mr.migrator]
recipe = mr.migrator
crawler-url=http://www.whitehouse.gov

or the same option can be overridden via the command line

$> bin/migrate --crawler:url=http://www.whitehouse.gov

some options require multiple lines within a buildout part. These can be overridden via the commandline by repeating the same argument e.g.

$> bin/migrate --crawler:ignore=\.mp3 --crawler:ignore=\.pdf

You use the commandline help to view the list of available options

$> bin/migrate --help

$> bin/mr.migrator --template1:debug

$> bin/mr.migrator --show-pipeline

$> bin/mr.migrator --show-pipeline > pipeline.cfg
$> {edit} pipeline.cfg
$> bin/mr.migrator --pipeline=pipeline.cfg

[mr.migrator]
recipe = mr.migrator
eggs = myblueprintpackage
pipeline = mypipeline.cfg

entry_points="""
      [z3c.autoinclude.plugin]
      target = transmogrify
      """,
      )

Help syntax for pipeline configuration

TODO

Mr.Migrator in Plone

*under development*

Mr.Migrator needs to be combined with a package providing a registered pipeline.

1. Install mr.migrator into your buildout

2. Install a package providing your pipeline such as funnelweb or collective.jsonmigrator

3. Go to the place in your site where you want to import content and select Actions > Mr.Migrate here (alternatively, open the URL @@mr.migrator)

4. Pick the pipeline you want from the drop down list

5. A form for filling in extra configuration for your pipeline will displayed (either autogenerated from the .cfg or a form designed by the pipeline author)

6. Click run

7. Popup a progress feedback dialog with a log of activity

TODO

• Finish autoform so works in all cases

• combine argsparse and autoform code

• do progress dialog

• hook point for packages to register form along with pipeline

• when no pipelines found: display help on where to find them and how to install them

Contributing

• Code repository: http://github.com/collective/mr.migrator

• Questions and comments to http://github.com/collective/mr.migrator/issues

• Report bugs at http://github.com/collective/mr.migrator/issues

Thanks

• Alex Clark - for the name

• Rok Garbas - the original code for the z3cform

• Dylan Jay - the original code of the commandline runner

1.2 (2017-05-15)

• Add Spanish translation. [macagua]

• Added more improvements about i18n support. [macagua]

• Added Buildout configuration file for Plone 4.3.x version. [macagua]

• Fix IOError: [Errno 2] No such file or directory: mr/migrator/autoinclude.zcml missing file. [macagua]

• Updated Buildout configuration file for Plone 3.3.x version. [macagua]

• Fix form problem with security hotfix 20160830 [agitator]

1.1 (2015-10-22)

• Remove browserlayer registration, so the mr.migrator doesn’t have to be installed to be used. Calling /@@mr.migrator is enough.

• Add uninstallation profile. [thet]

• Allow any pipeline configuration to be imported through the web at the @@mr.migrator view. Previously, only those pipelines were shown, which had a plone.app.transmogrifier.atschemaupdater blueprint included. Now any pipelines can be used, e.g. those which handle only Dexterity objects. [thet]

• Pep8. [thet]

1.0.1 (2012-09-18)

• put back in manual zcml loading [djay]

1.0 (2012-04-28)

• fix setup.py issue breaking buildout [djay]

• fixed NameError: global name ‘cparser’ is not defined, issue #6 [gborelli]

• fix default profile [toutpt]

• fix import error [toutpt]

1.0b11 (2012-04-25)

• support pipeline includes [djay]

1.0b10 (2012-01-23)

• Revert portion of ec799dcd3d, causing ConfigurationErrors [aclark]

1.0b9 (2012-01-23)

• Fix brown bag release (bad MANIFEST) [aclark]

1.0b8 (2012-01-23)

• Fix brown bag release (missing zcml) [aclark]

1.0b7 (2012-01-23)

• Look for pipeline.cfg in cwd [aclark]

1.0b6 (2012-01-23)

• Fix brown bag release [aclark]

1.0b5 (2012-01-22)

• Bug fix: support for –zcml arguments [aclark]

1.0b4 (2011-08-14)

• Bug fix: ZCML config [aclark]

1.0b3 (2011-08-14)

• Bug fix: ZCML config [aclark]

1.0b2 (2011-08-14)

• Bug fix: Python 2.4 compat [aclark]

1.0b1 (2011-06-29)

• allow way of running zcml for blueprint packages in commandline [djay]

• split out commandline runner from funnelweb [djay]

• add start of form for running transmogrifier inside plone [djay]