collective.transmogrifier 1.3

Pipelines

To transmogrify, or import and convert non-plone content, you simply define a pipeline. Pipe sections, the equivalent of parts in a buildout, are slotted together into a processing pipe. To slot sections together, you define a configuration file, define named sections, and a main pipeline definition that names the sections in order (one section per line):

>>> exampleconfig = """\
... [transmogrifier]
... pipeline =
...     section 1
...     section 2
...     section 3
...
... [section 1]
... blueprint = collective.transmogrifier.tests.examplesource
... size = 5
...
... [section 2]
... blueprint = collective.transmogrifier.tests.exampletransform
...
... [section 3]
... blueprint = collective.transmogrifier.tests.exampleconstructor
... """

As you can see this is also very similar to how you construct WSGI pipelines using paster. The format of the configuration files is defined by the Python ConfigParser module, with extensions that we’ll describe later. At minimum, at least the transmogrifier section with an empty pipeline is required:

>>> mimimalconfig = """\
... [transmogrifier]
... pipeline =
... """

Transmogrifier can load these configuration files either by looking them up in a registry or by loading them from a python package.

You register transmogrifier configurations using the registerConfig directive in the http://namespaces.plone.org/transmogrifier namespace, together with a name, and optionally a title and description:

<configure
    xmlns="http://namespaces.zope.org/zope"
    xmlns:transmogrifier="http://namespaces.plone.org/transmogrifier"
    i18n_domain="collective.transmogrifier">

<transmogrifier:registerConfig
    name="exampleconfig"
    title="Example pipeline configuration"
    description="This is an example pipeline configuration"
    configuration="example.cfg"
    />

</configure>

You can then tell transmogrifier to load the ‘exampleconfig’ configuration. To load configuration files directly from a python package, name the package and the configuration file separated by a colon, such as ‘collective.transmogrifier.tests:exampleconfig.cfg’.

Registering files with the transmogrifier registry allows other uses, such as listing available configurations in a user interface, together with the registered description. Loading files directly let’s you build reusable libraries of configuration files more quickly though.

In this document we’ll use the shorthand registerConfig to register example configurations:

>>> registerConfig(u'collective.transmogrifier.tests.exampleconfig',
...                exampleconfig)

def __iter__(self):
    # Before iteration
    # You can do initialisation here

    for item in self.previous
        # Iteration itself
        # You could process the items, take notes, inject additional
        # items based on the current item in the pipe or manipulate portal
        # content created by previous items
        yield item

    # After iteration
    # The section still has control here and could inject additional
    # items, manipulate all portal content created by the pipeline,
    # or clean up after itself.

from zope.annotation.interfaces import IAnnotations

MYKEY = 'foo.bar.baz'

def __init__(self, transmogrifier, name, options, previous):
    self.storage = IAnnotations(transmogrifier).setdefault(MYKEY, {})
    self.storage.setdefault('spam', 0)
    ...

def __iter__(self):
    ...
    self.storage['spam'] += 1
    ...

GenericSetup import integration

To ease running a transmogrifier pipeline during site configuration, a generic import step for GenericSetup is included.

The import step looks for a file named transmogrifier.txt and reads pipeline configuration names from this file, one name per line. Empty lines and lines starting with a # (hash mark) are skipped. These pipelines are then executed in the same order as they are found in the file.

This means that if you want to run one or more pipelines as part of a GenericSetup profile, all you have to do is name these pipelines in a file named transmogrifier.txt in your profile directory.

The GenericSetup import context is stored on the transmogrifier as an annotation:

from collective.transmogrifier.genericsetup import IMPORT_CONTEXT
from zope.annotation.interfaces import IAnnotations

def __init__(self, transmogrifier, name, options, previous):
    self.import_context = IAnnotations(transmogrifier)[IMPORT_CONTEXT]

This will of course prevent your code from running outside the generic setup import context.

Constructor section

A constructor pipeline section is the heart of a transmogrifier content import pipeline. It constructs Plone content based on the items it processes. The constructor section blueprint name is collective.transmogrifier.sections.constructor. Constructor sections do only one thing, they construct new content. No schema changes are made. Also, constructors create content without restrictions, no security checks or containment constraints are checked.

Construction needs 2 pieces of information: the path to the item (including the id for the new item itself) and it’s portal type. To determine both of these, the constructor section inspects each item and looks for 2 keys, as described below. Any item missing any of these 2 pieces will be skipped. Similarly, items with a path for a container or type that doesn’t exist will be skipped as well; make sure that these containers are constructed beforehand. Because a constructor section will only construct new objects, if an object with the same path already exists, the item will also be skipped.

For the object path, it’ll look (in order) for _collective.transmogrifier.sections.constructor_[sectionname]_path, _collective.transmogrifier.sections.constructor_path, _[sectionname]_path, and _path, where [sectionname] is replaced with the name given to the current section. This allows you to target the right section precisely if needed. Alternatively, you can specify what key to use for the path by specifying the path-key option, which should be a list of keys to try (one key per line, use a re: or regexp: prefix to specify regular expressions).

For the portal type, use the type-key option to specify a set of keys just like path-key. If omitted, the constructor will look for _collective.transmogrifier.sections.constructor_[sectionname]_type, _collective.transmogrifier.sections.constructor_type, _[sectionname]_type, _type, portal_type and Type (in that order, with [sectionname] replaced).

Unicode paths will be encoded to ASCII. Using the path and type, a new object will be constructed using invokeFactory; nothing else is done. Paths are always interpreted as relative to the context object, with the last path segment being the id of the object to create.

By default the constructor section will log a warning if the container for the item is missing and the item can’t be constructed. However if you add a required = True key to the constructor section it will instead raise a KeyError.

>>> import pprint
>>> constructor = """
... [transmogrifier]
... pipeline =
...     contentsource
...     constructor
...     printer
...
... [contentsource]
... blueprint = collective.transmogrifier.sections.tests.contentsource
...
... [constructor]
... blueprint = collective.transmogrifier.sections.constructor
...
... [printer]
... blueprint = collective.transmogrifier.sections.tests.pprinter
... """
>>> registerConfig(u'collective.transmogrifier.sections.tests.constructor',
...                constructor)
>>> transmogrifier(u'collective.transmogrifier.sections.tests.constructor')
[('_path', '/spam/eggs/foo'), ('_type', 'FooType')]
[('_path', '/foo'), ('_type', 'FooType')]
[('_path', u'/unicode/encoded/to/ascii'), ('_type', 'FooType')]
[('_path', 'not/existing/bar'),
 ('_type', 'BarType'),
 ('title', 'Should not be constructed, not an existing path')]
[('_path', '/spam/eggs/existing'),
 ('_type', 'FooType'),
 ('title', 'Should not be constructed, an existing object')]
[('_path', '/spam/eggs/incomplete'),
 ('title', 'Should not be constructed, no type')]
[('_path', '/spam/eggs/nosuchtype'),
 ('_type', 'NonExisting'),
 ('title', 'Should not be constructed, not an existing type')]
[('_path', 'spam/eggs/changedByFactory'),
 ('_type', 'FooType'),
 ('title', 'Factories are allowed to change the id')]
>>> pprint.pprint(plone.constructed)
(('spam/eggs', 'foo', 'FooType'),
 ('', 'foo', 'FooType'),
 ('unicode/encoded/to', 'ascii', 'FooType'),
 ('spam/eggs', 'changedByFactory', 'FooType'))

>>> constructor = """
... [transmogrifier]
... pipeline =
...     contentsource
...     constructor
...     printer
...
... [contentsource]
... blueprint = collective.transmogrifier.sections.tests.contentsource
...
... [constructor]
... blueprint = collective.transmogrifier.sections.constructor
... required = True
...
... [printer]
... blueprint = collective.transmogrifier.sections.tests.pprinter
... """
>>> registerConfig(u'collective.transmogrifier.sections.tests.constructor2',
...                constructor)
>>> try:
...     transmogrifier(u'collective.transmogrifier.sections.tests.constructor2')
...     raise AssertionError("Required constructor did not raise an error for missing folder")
... except KeyError:
...     pass
[('_path', '/spam/eggs/foo'), ('_type', 'FooType')]
[('_path', '/foo'), ('_type', 'FooType')]
[('_path', u'/unicode/encoded/to/ascii'), ('_type', 'FooType')]

Folders section

The collective.transmogrifier.sections.constructor blueprint can construct new content, based on a type (_type key) and a path (_path key). However, it will bail if it is asked to create an item for which the parent folder does not exist.

One way to work around this is to ensure that the folders already exist, for example by sending the instruction to construct them through the pipeline before any contents of that folder. This requires sorted input, of course.

Alternatively, you can use the collective.transmogrifier.sections.folders blueprint. This will look at the path of each incoming item and construct parent folders if needed. This implies that all folders (that do not yet exist), are of the same type. That type defaults to Folder, although you can supply an alternative type. The folder will be created without an id only, but a subsequent schema updated section for a subsequent item may have the opportunity to update it (but not change its type.)

This blueprint can take the following options, all of the optional:

path-key

The name of the key holding the path. This defaults to the same semantics as those used for the constructor section. Just use _path and you’ll be OK.

new-type-key

The type key to use when inserting a new item in the pipeline to create folders. The default is _type. Change it if you need to target a specific constructor section.

new-path-key

The path key to use when inserting a new item in the pipeline to create folders. The default is to use the same as the incoming path key. Change it if you need to target a specific constructor section.

folder-type

The name of the portal type to use for new folders. Defaults to Folder, which is the default folder type in CMF and Plone.

cache

By default, the section will keep a cache in memory of each folder it has checked (and possibly created) to know whether it already exists. This saves a lot of traversal, especially if you have many items under a particular folder. This will use a small amount of memory. If you have millions of objects, you can trade memory for speed by setting this option to false.

Here is how it might look by default:

>>> import pprint
>>> constructor = """
... [transmogrifier]
... pipeline =
...     contentsource
...     folders
...     printer
...
... [contentsource]
... blueprint = collective.transmogrifier.sections.tests.folderssource
...
... [folders]
... blueprint = collective.transmogrifier.sections.folders
...
... [printer]
... blueprint = collective.transmogrifier.sections.tests.pprinter
... """
>>> registerConfig(u'collective.transmogrifier.sections.tests.folders',
...                constructor)
>>> transmogrifier(u'collective.transmogrifier.sections.tests.folders')
[('_path', '/foo'), ('_type', 'Document')]
[('_path', '/existing/foo'), ('_type', 'Document')]
[('_path', '/nonexisting'), ('_type', 'Folder')]
[('_path', '/nonexisting/alpha'), ('_type', 'Folder')]
[('_path', '/nonexisting/alpha/foo'), ('_type', 'Document')]
[('_path', '/nonexisting/beta'), ('_type', 'Folder')]
[('_path', '/nonexisting/beta/foo'), ('_type', 'Document')]
[('_type', 'Document')]
[('_folders_path', '/delta'), ('_type', 'Folder')]
[('_folders_path', '/delta/foo'), ('_type', 'Document')]

To specify alternate types and keys, we can do something like this:

>>> import pprint
>>> constructor = """
... [transmogrifier]
... pipeline =
...     contentsource
...     folders
...     printer
...
... [contentsource]
... blueprint = collective.transmogrifier.sections.tests.folderssource
...
... [folders]
... blueprint = collective.transmogrifier.sections.folders
... folder-type = My Folder
... new-type-key = '_folderconstructor_type
... new-path-key = '_folderconstructor_path
...
... [printer]
... blueprint = collective.transmogrifier.sections.tests.pprinter
... """
>>> registerConfig(u'collective.transmogrifier.sections.tests.folders2',
...                constructor)
>>> transmogrifier(u'collective.transmogrifier.sections.tests.folders2')
[('_path', '/foo'), ('_type', 'Document')]
[('_path', '/existing/foo'), ('_type', 'Document')]
[("'_folderconstructor_path", '/nonexisting'),
 ("'_folderconstructor_type", 'My Folder')]
[("'_folderconstructor_path", '/nonexisting/alpha'),
 ("'_folderconstructor_type", 'My Folder')]
[('_path', '/nonexisting/alpha/foo'), ('_type', 'Document')]
[("'_folderconstructor_path", '/nonexisting/beta'),
 ("'_folderconstructor_type", 'My Folder')]
[('_path', '/nonexisting/beta/foo'), ('_type', 'Document')]
[('_type', 'Document')]
[("'_folderconstructor_path", '/delta'),
 ("'_folderconstructor_type", 'My Folder')]
[('_folders_path', '/delta/foo'), ('_type', 'Document')]

Codec section

A codec pipeline section lets you alter the character encoding of item values, allowing you to recode text from and to unicode and any of the codecs supported by python. The codec section blueprint name is collective.transmogrifier.sections.codec.

What values to recode is determined by the keys option, which takes a set of newline-separated key names. If a key name starts with re: or regexp: it is treated as a regular expression instead.

The optional from and to options determine what codecs values are recoded from and to. Both these values default to unicode, meaning no translation. If either option is set to default, the current default encoding of the Plone site is used.

To deal with possible encoding errors, you can set the error handler of both the from and to codecs separately with the from-error-handler and to-error-handler options, respectively. These default to strict, but can be set to any error handler supported by python, including replace and ignore.

Also optional is the condition option, which lets you specify a TALES expression that when evaluating to False will prevent any en- or decoding from happening. The condition is evaluated for every matched key.

>>> codecs = """
... [transmogrifier]
... pipeline =
...     source
...     decode-all
...     encode-id
...     encode-title
...     printer
...
... [source]
... blueprint = collective.transmogrifier.sections.tests.samplesource
... encoding = utf8
...
... [decode-all]
... blueprint = collective.transmogrifier.sections.codec
... keys = re:.*
... from = utf8
...
... [encode-id]
... blueprint = collective.transmogrifier.sections.codec
... keys = id
... to = ascii
...
... [encode-title]
... blueprint = collective.transmogrifier.sections.codec
... keys = title
... to = ascii
... to-error-handler = backslashreplace
... condition = python:'Brand' not in item['title']
...
... [printer]
... blueprint = collective.transmogrifier.sections.tests.pprinter
... """
>>> registerConfig(u'collective.transmogrifier.sections.tests.codecs',
...                codecs)
>>> transmogrifier(u'collective.transmogrifier.sections.tests.codecs')
[('id', 'foo'), ('status', u'\u2117'), ('title', 'The Foo Fighters \\u2117')]
[('id', 'bar'), ('status', u'\u2122'), ('title', u'Brand Chocolate Bar \u2122')]
[('id', 'monty-python'), ('status', u'\xa9'), ('title', "Monty Python's Flying Circus \\xa9")]

The condition expression has access to the following:

Inserter section

An inserter pipeline section lets you define a key and value to insert into pipeline items. The inserter section blueprint name is collective.transmogrifier.sections.inserter.

A inserter section takes a key and a value TALES expression. These expressions are evaluated to generate the actual key-value pair that gets inserted. You can also specify an optional condition option; if given, the key only gets inserted when the condition, which is also a TALES is true.

Because the inserter value expression has access to the original item, it could even be used to change existing item values. Just target an existing key, pull out the original value in the value expression and return a modified version.

>>> inserter = """
... [transmogrifier]
... pipeline =
...     source
...     simple-insertion
...     expression-insertion
...     transform-id
...     printer
...
... [source]
... blueprint = collective.transmogrifier.sections.tests.rangesource
... size = 3
...
... [simple-insertion]
... blueprint = collective.transmogrifier.sections.inserter
... key = string:foo
... value = string:bar (inserted into "${item/id}" by the "$name" section)
...
... [expression-insertion]
... blueprint = collective.transmogrifier.sections.inserter
... key = python:'foo-%s' % item['id'][-2:]
... value = python:int(item['id'][-2:]) * 15
... condition = python:int(item['id'][-2:])
...
... [transform-id]
... blueprint = collective.transmogrifier.sections.inserter
... key = string:id
... value = string:foo-${item/id}
...
... [printer]
... blueprint = collective.transmogrifier.sections.tests.pprinter
... """
>>> registerConfig(u'collective.transmogrifier.sections.tests.inserter',
...                inserter)
>>> transmogrifier(u'collective.transmogrifier.sections.tests.inserter')
[('foo', 'bar (inserted into "item-00" by the "simple-insertion" section)'), ('id', 'foo-item-00')]
[('foo', 'bar (inserted into "item-01" by the "simple-insertion" section)'), ('foo-01', 15), ('id', 'foo-item-01')]
[('foo', 'bar (inserted into "item-02" by the "simple-insertion" section)'), ('foo-02', 30), ('id', 'foo-item-02')]

The key, value and condition expressions have access to the following:

Condition section

A condition pipeline section lets you selectively discard items from the pipeline. The condition section blueprint name is collective.transmogrifier.sections.condition.

A condition section takes a condition TALES expression. When this expression when matched against the current item is True, the item is yielded to the next pipe section, otherwise it is not:

>>> condition = """
... [transmogrifier]
... pipeline =
...     source
...     condition
...     printer
...
... [source]
... blueprint = collective.transmogrifier.sections.tests.rangesource
... size = 5
...
... [condition]
... blueprint = collective.transmogrifier.sections.condition
... condition = python:int(item['id'][-2:]) > 2
...
... [printer]
... blueprint = collective.transmogrifier.sections.tests.pprinter
... """
>>> registerConfig(u'collective.transmogrifier.sections.tests.condition',
...                condition)
>>> transmogrifier(u'collective.transmogrifier.sections.tests.condition')
[('id', 'item-03')]
[('id', 'item-04')]

The condition expression has access to the following:

As condition sections skip items in the pipeline, they should not be used inside a splitter section!

Manipulator section

A manipulator pipeline section lets you copy, move or discard keys from the pipeline. The manipulator section blueprint name is collective.transmogrifier.sections.manipulator.

A manipulator section will copy keys when you specify a set of keys to copy, and an expression to determine what to copy these to. These are the keys and destination options.

The keys option is a set of key names, one on each line; keynames starting with re: or regexp: are treated as regular expresions. The destination expression is a TALES expression that can access not only the item, but also the matched key and, if a regular expression was used, the match object.

If a delete option is specified, it is also interpreted as a set of keys, like the keys option. These keys will be deleted from the item; if used together with the keys and destination options, keys will be renamed instead of copied.

Also optional is the condition option, which lets you specify a TALES expression that when evaluating to False will prevent any manipulation from happening. The condition is evaluated for every matched key.

>>> manipulator = """
... [transmogrifier]
... pipeline =
...     source
...     copy
...     rename
...     delete
...     printer
...
... [source]
... blueprint = collective.transmogrifier.sections.tests.samplesource
...
... [copy]
... blueprint = collective.transmogrifier.sections.manipulator
... keys =
...     title
...     id
... destination = string:$key-copy
...
... [rename]
... blueprint = collective.transmogrifier.sections.manipulator
... keys = re:([^-]+)-copy$
... destination = python:'%s-duplicate' % match.group(1)
... delete = ${rename:keys}
...
... [delete]
... blueprint = collective.transmogrifier.sections.manipulator
... delete = status
...
... [printer]
... blueprint = collective.transmogrifier.sections.tests.pprinter
... """
>>> registerConfig(u'collective.transmogrifier.sections.tests.manipulator',
...                manipulator)
>>> transmogrifier(u'collective.transmogrifier.sections.tests.manipulator')
[('id', 'foo'),
 ('id-duplicate', 'foo'),
 ('title', u'The Foo Fighters \u2117'),
 ('title-duplicate', u'The Foo Fighters \u2117')]
[('id', 'bar'),
 ('id-duplicate', 'bar'),
 ('title', u'Brand Chocolate Bar \u2122'),
 ('title-duplicate', u'Brand Chocolate Bar \u2122')]
[('id', 'monty-python'),
 ('id-duplicate', 'monty-python'),
 ('title', u"Monty Python's Flying Circus \xa9"),
 ('title-duplicate', u"Monty Python's Flying Circus \xa9")]

The destination expression has access to the following:

Splitter section

A splitter pipeline section lets you branch a pipeline into 2 or more sub-pipelines. The splitter section blueprint name is collective.transmogrifier.sections.splitter.

A splitter section takes 2 or more pipeline definitions, and sends the items from the previous section through each of these sub-pipelines, each with it’s own copy [*] of the items:

>>> emptysplitter = """
... [transmogrifier]
... pipeline =
...     source
...     splitter
...     printer
...
... [source]
... blueprint = collective.transmogrifier.sections.tests.rangesource
... size = 3
...
... [splitter]
... blueprint = collective.transmogrifier.sections.splitter
... pipeline-1 =
... pipeline-2 =
...
... [printer]
... blueprint = collective.transmogrifier.sections.tests.pprinter
... """
>>> registerConfig(u'collective.transmogrifier.sections.tests.emptysplitter',
...                emptysplitter)
>>> transmogrifier(u'collective.transmogrifier.sections.tests.emptysplitter')
[('id', 'item-00')]
[('id', 'item-00')]
[('id', 'item-01')]
[('id', 'item-01')]
[('id', 'item-02')]
[('id', 'item-02')]

Although the pipeline definitions in the splitter are empty, we end up with 2 copies of every item in the pipeline as both splitter pipelines get to process a copy. Splitter pipelines are defined by options starting with pipeline-.

Normally you’ll use conditions to identify items for each sub-pipe, making the splitter the pipeline equivalent of an if/elif statement. Conditions are optional and use the pipeline option name plus -condition:

>>> evenoddsplitter = """
... [transmogrifier]
... pipeline =
...     source
...     splitter
...     printer
...
... [source]
... blueprint = collective.transmogrifier.sections.tests.rangesource
... size = 3
...
... [splitter]
... blueprint = collective.transmogrifier.sections.splitter
... pipeline-even-condition = python:int(item['id'][-2:]) % 2
... pipeline-even = even-section
... pipeline-odd-condition = not:${splitter:pipeline-even-condition}
... pipeline-odd = odd-section
...
... [odd-section]
... blueprint = collective.transmogrifier.sections.inserter
... key = string:even
... value = string:The even pipe
...
... [even-section]
... blueprint = collective.transmogrifier.sections.inserter
... key = string:odd
... value = string:The odd pipe
...
... [printer]
... blueprint = collective.transmogrifier.sections.tests.pprinter
... """
>>> registerConfig(u'collective.transmogrifier.sections.tests.evenodd',
...                evenoddsplitter)
>>> transmogrifier(u'collective.transmogrifier.sections.tests.evenodd')
[('even', 'The even pipe'), ('id', 'item-00')]
[('id', 'item-01'), ('odd', 'The odd pipe')]
[('even', 'The even pipe'), ('id', 'item-02')]

Conditions are expressed as TALES statements, and have access to:

Savepoint section

A savepoint pipeline section commits a savepoint every so often, which has a side-effect of freeing up memory. The savepoint section blueprint name is collective.transmogrifier.sections.savepoint.

A savepoint section takes an optional every option, which defaults to 1000; a savepoint is committed every every items passing through the pipe. A savepoint section doesn’t alter the items in any way:

>>> savepoint = """
... [transmogrifier]
... pipeline =
...     source
...     savepoint
...
... [source]
... blueprint = collective.transmogrifier.sections.tests.rangesource
... size = 10
...
... [savepoint]
... blueprint = collective.transmogrifier.sections.savepoint
... every = 3
... """
>>> registerConfig(u'collective.transmogrifier.sections.tests.savepoint',
...                savepoint)

We’ll show savepoints being committed by overriding transaction.savepoint:

>>> import transaction
>>> original_savepoint = transaction.savepoint
>>> counter = [0]
>>> def test_savepoint(counter=counter, *args, **kw):
...     counter[0] += 1
>>> transaction.savepoint = test_savepoint
>>> transmogrifier(u'collective.transmogrifier.sections.tests.savepoint')
>>> transaction.savepoint = original_savepoint
>>> counter[0]
3

CSV source section

A CSV source pipeline section lets you create pipeline items from CSV files. The CSV source section blueprint name is collective.transmogrifier.sections.csvsource.

A CSV source section will load the CSV file named in the filename option, and will yield an item for each line in the CSV file. It’ll use the first line of the CSV file to determine what keys to use, or you can specify a fieldnames option to specify the key names.

The filename option may be an absolute path, or a package reference, e.g. my.package:foo/bar.csv.

By default the CSV file is assumed to use the Excel CSV dialect, but you can specify any dialect supported by the python csv module if you specify it with the dialect option.

>>> import tempfile
>>> tmp = tempfile.NamedTemporaryFile('w+', suffix='.csv')
>>> tmp.write('\r\n'.join("""\
... foo,bar,baz
... first-foo,first-bar,first-baz
... second-foo,second-bar,second-baz
... """.splitlines()))
>>> tmp.flush()
>>> csvsource = """
... [transmogrifier]
... pipeline =
...     csvsource
...     printer
...
... [csvsource]
... blueprint = collective.transmogrifier.sections.csvsource
... filename = %s
...
... [printer]
... blueprint = collective.transmogrifier.sections.tests.pprinter
... """ % tmp.name
>>> registerConfig(u'collective.transmogrifier.sections.tests.csvsource.file',
...                csvsource)
>>> transmogrifier(u'collective.transmogrifier.sections.tests.csvsource.file')
[('bar', 'first-bar'), ('baz', 'first-baz'), ('foo', 'first-foo')]
[('bar', 'second-bar'), ('baz', 'second-baz'), ('foo', 'second-foo')]

>>> transmogrifier(u'collective.transmogrifier.sections.tests.csvsource.file',
...                csvsource=dict(fieldnames='monty spam eggs'))
[('eggs', 'baz'), ('monty', 'foo'), ('spam', 'bar')]
[('eggs', 'first-baz'), ('monty', 'first-foo'), ('spam', 'first-bar')]
[('eggs', 'second-baz'), ('monty', 'second-foo'), ('spam', 'second-bar')]

Here is the same example, loading a file from a package instead:

>>> csvsource = """
... [transmogrifier]
... pipeline =
...     csvsource
...     printer
...
... [csvsource]
... blueprint = collective.transmogrifier.sections.csvsource
... filename = collective.transmogrifier.tests:sample.csv
...
... [printer]
... blueprint = collective.transmogrifier.sections.tests.pprinter
... """
>>> registerConfig(u'collective.transmogrifier.sections.tests.csvsource.package',
...                csvsource)
>>> transmogrifier(u'collective.transmogrifier.sections.tests.csvsource.package')
[('bar', 'first-bar'), ('baz', 'first-baz'), ('foo', 'first-foo')]
[('bar', 'second-bar'), ('baz', 'second-baz'), ('foo', 'second-foo')]

>>> transmogrifier(u'collective.transmogrifier.sections.tests.csvsource.package',
...                csvsource=dict(fieldnames='monty spam eggs'))
[('eggs', 'baz'), ('monty', 'foo'), ('spam', 'bar')]
[('eggs', 'first-baz'), ('monty', 'first-foo'), ('spam', 'first-bar')]
[('eggs', 'second-baz'), ('monty', 'second-foo'), ('spam', 'second-bar')]

1.3 (2011-03-17)

• Added the GenericSetup import context as an annotation to the transmogrifier. [elro]

• Added a logger to log the value of a particular key for all items. Handy when debugging, you can see which path is failing, and good if you want to show progress in a long import. [regebro]

• Added a breakpoint section to break on a particular expression, which is handy for debugging. [regebro]

1.2 (2010-03-30)

• Bug fix: the constructor promises to encode paths to ASCII, but failed to do so. Thanks to gyst for finding the discrepancy. [mj]

1.1 (2010-03-17)

• Allow the CSV source to load its file from a package as well as from an absolute or relative file path. To load from a package, pass package.name:filename.csv to the filename option. [optilude]

• Add CMF 2.2/Plone 4 compatibility for the content constructor [optilude]

• Use an explicit provides attribute to register the transmogrifier adapter. Fixes the “Missing ‘provides’ attribute” errors when loading with zope.annotation installed. [mj]

• Add a required flag to the content constructor, which causes it to raise a KeyError if the container where to construct the new item doesn’t exist. [regebro]

• Add an optional condition to the manipulator section. [regebro]

1.0 (2009-08-07)

• Initial transmogrifier architecture. [mj]