CSV import/export for Archetypes or other contents (via plugins for the later). Created by Makina Corpus.
Project description
Introduction
Allows to export and import AT objects contents and hierarchy.
It supports also other contents or objects via plugins (ZCA). This part mainly targeted for a developper audiance, so see interfaces.py / adapters.py.
We have plans to integrate the export/import code with transmogrifier, [ also ] but we have no ETA For now.
Repository: git
A Makina Corpus add-on
Products.csvreplicata Installation
To install Products.csvreplicata into the global Python environment (or a workingenv), using a traditional Zope 2 instance, you can do this:
When you’re reading this you have probably already run easy_install Products.csvreplicata. Find out how to install setuptools (and EasyInstall) here: http://peak.telecommunity.com/DevCenter/EasyInstall
If you are using Zope 2.9 (not 2.10), get pythonproducts and install it via:
python setup.py install --home /path/to/instanceinto your Zope instance.
Create a file called Products.csvreplicata-configure.zcml in the /path/to/instance/etc/package-includes directory. The file should only contain this:
<include package="Products.csvreplicata" />
Alternatively, if you are using zc.buildout and the plone.recipe.zope2instance recipe to manage your project, you can do this:
Add Products.csvreplicata to the list of eggs to install, e.g.
[buildout]
...
eggs =
...
Products.csvreplicata
* Tell the plone.recipe.zope2instance recipe to install a ZCML slug
[instance]
recipe = plone.recipe.zope2instance
...
zcml =
Products.csvreplicata
* Re-run buildout, e.g. with:
$ ./bin/buildout
You can skip the ZCML slug if you are going to explicitly include the package from another package’s configure.zcml file.
Because its top level Python namespace package is called Products, this package can also be installed in Zope 2 as an old style Zope 2 Product.
For that, move (or symlink) the csvreplicata folder of this project (Products.csvreplicata/Products/csvreplicata) into the Products directory of the Zope instance it has to be installed for, and restart the server.
You can also skip the ZCML slug if you install this package the Zope 2 Product way.
Detailled documentation
Interfaces
import interfaces and classes
>>> from zope.interface.verify import verifyClass >>> from zope.interface import implements >>> from Products.csvreplicata.handlers.base import CSVdefault >>> from Products.csvreplicata.handlers.file import CSVFile >>> from Products.csvreplicata.interfaces import ICSVDefault, ICSVFile
Verify implementation
>>> verifyClass(ICSVDefault, CSVdefault) True >>> verifyClass(ICSVFile, CSVFile) True
Export / Import in plain format
Export
here we export folders and documents
>>> self.setRoles(['Manager'])
>>> id=self.folder.invokeFactory('Document' , id='doc1' , title="Document 1")
>>> id=self.folder.invokeFactory('Document' , id='doc2' , title="Document 2")
>>> id=self.folder.invokeFactory('News Item' , id='news1' , title="news 'super' 3")
>>> id=self.folder.invokeFactory('Document' , id='doc4' , title="Document 4")
>>> id=self.folder.invokeFactory('Folder' , id='sub1' , title="mytest")
>>> id=self.folder.sub1.invokeFactory('Document' , id='doc11' , title="Document 1 du dossier 1")
>>> id=self.folder.sub1.invokeFactory('News Item', id='news21' , title="news 1 du dossier 1")
>>> id=self.folder.sub1.invokeFactory('Document' , id='doc31' , title="Document 2 du dossier 1")
>>> self.portal.portal_csvreplicatatool.getPlainFormat()
False
>>> self.portal.portal_csvreplicatatool.getEncoding()
'UTF-8'
>>> self.portal.portal_csvreplicatatool.getDelimiter()
';'
>>> self.portal.portal_csvreplicatatool.getStringdelimiter()
'"'
>>> self.portal.portal_csvreplicatatool.replicabletypes = {'Document': ['default'], 'Folder': ['default'], 'News': ['Default']}
>>> from Products.csvreplicata.interfaces import Icsvreplicata
>>> repl = Icsvreplicata(self.folder)
>>> print repl.csvexport(exportable_content_types=['Document', 'Folder', 'News Item'], depth=0).getvalue()
"/plone/Members/test_user_1_";"..."
"parent";"id";"type";"title";"description";"text";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"Parent folder";"Identifier";"Content type";"Title";"label_summary";"label_body_text";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"";"doc1";"Document";"Document 1";"";"";"simple_publication_workflow";"private";"";""
"";"doc2";"Document";"Document 2";"";"";"simple_publication_workflow";"private";"";""
"parent";"id";"type";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"Parent folder";"Identifier";"Content type";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"";"news1";"News Item";"simple_publication_workflow";"private"
"parent";"id";"type";"title";"description";"text";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"Parent folder";"Identifier";"Content type";"Title";"label_summary";"label_body_text";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"";"doc4";"Document";"Document 4";"";"";"simple_publication_workflow";"private"
"parent";"id";"type";"title";"description";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"Parent folder";"Identifier";"Content type";"Title";"label_description";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"";"sub1";"Folder";"mytest";"";"simple_publication_workflow";"private"
"parent";"id";"type";"title";"description";"text";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"Parent folder";"Identifier";"Content type";"Title";"label_summary";"label_body_text";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"sub1";"doc11";"Document";"Document 1 du dossier 1";"";"";"simple_publication_workflow";"private"
"parent";"id";"type";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"Parent folder";"Identifier";"Content type";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"sub1";"news21";"News Item";"simple_publication_workflow";"private"
"parent";"id";"type";"title";"description";"text";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"Parent folder";"Identifier";"Content type";"Title";"label_summary";"label_body_text";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"sub1";"doc31";"Document";"Document 2 du dossier 1";"";"";"simple_publication_workflow";"private"
<BLANKLINE>
This output is not that good to deal with CSV apis, we will try to export it to a flat structure.
>>> from Products.csvreplicata import replicator >>> self.portal.portal_csvreplicatatool.setPlainFormat(True) >>> self.portal.portal_csvreplicatatool.getPlainFormat() True >>> repl = Icsvreplicata(self.folder) >>> print Icsvreplicata(self.folder).csvexport(exportable_content_types=['Document', 'Folder', 'News Item'], depth=0).getvalue() # doctest: +ELLIPSIS "startpoint";"replicata_export_date";"parent";"id";"type";"title";"description";... "Start point";"Export Date";"Parent folder";"Identifier";"Content type";"Title";... "/plone/Members/test_user_1_";"...";"";"doc1";"Document";"Document 1";... "/plone/Members/test_user_1_";"...";"";"doc2";"Document";"Document 2";... "/plone/Members/test_user_1_";"...";"";"news1";"News Item";... "/plone/Members/test_user_1_";"...";"";"doc4";"Document";"Document 4";... "/plone/Members/test_user_1_";"...";"";"sub1";"Folder";"mytest";... "/plone/Members/test_user_1_";"...";"sub1";"doc11";"Document";"Document 1 du dossier 1";... "/plone/Members/test_user_1_";"...";"sub1";"news21";"News Item";... "/plone/Members/test_user_1_";"...";"sub1";"doc31";"Document";"Document 2 du dossier 1";... <BLANKLINE>
Redo the export but with a plugin that find the title on the object.
>>> from Products.csvreplicata import adapters
>>> class CustomExporter(adapters.CSVReplicataExportImportPluginAbstract):
... def __init__(self, *args, **kwargs):
... adapters.CSVReplicataExportImportPluginAbstract.__init__(self, *args, **kwargs)
... self.ids.append('title')
... def fill_values(self, row, row_ids):
... """."""
... for id in row_ids:
... if id.replace(self.prefix, '') in self.ids:
... index = row_ids.index(id)
... if index < len(row):
... row[index] = self.context.Title()
... def set_values(self, row, row_ids):
... """."""
... print "plugin.setValue called with %s <-> %s" % (row, row_ids)
...
>>> provideAdapter(CustomExporter, (interfaces.Icsvreplicata, zope.interface.Interface), interfaces.ICSVReplicataExportImportPlugin, name ='fooplugin' )
>>> from csv import DictReader
>>> content = Icsvreplicata(self.folder).csvexport(exportable_content_types=['Document', 'Folder', 'News Item'], depth=None).getvalue()
>>> items = [item for item in DictReader(StringIO(content), delimiter=";", quotechar='"')]
>>> keys = items[0].keys();keys.sort()
As we cant predict order of the keys, doing some magic to order them before testing.
>>> pprint([[(key, item[key]) for key in keys if not 'date' in key]for item in items], width=130) # doctest: +REPORT_NDIFF
[[('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain',
'ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain'),
('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state',
'ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state'),
('ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title',
'ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title'),
('description', 'label_...'),
('id', 'Identifier'),
('parent', 'Parent folder'),
('startpoint', 'Start point'),
('text', 'label_body_text'),
('title', 'Title'),
('type', 'Content type')],
[('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain', 'simple_publication_workflow'),
('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state', 'private'),
('ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title', 'Document 1'),
('description', ''),
('id', 'doc1'),
('parent', ''),
('startpoint', '/plone/Members/test_user_1_'),
('text', ''),
('title', 'Document 1'),
('type', 'Document')],
[('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain', 'simple_publication_workflow'),
('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state', 'private'),
('ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title', 'Document 2'),
('description', ''),
('id', 'doc2'),
('parent', ''),
('startpoint', '/plone/Members/test_user_1_'),
('text', ''),
('title', 'Document 2'),
('type', 'Document')],
[('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain', 'simple_publication_workflow'),
('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state', 'private'),
('ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title', "news 'super' 3"),
('description', ''),
('id', 'news1'),
('parent', ''),
('startpoint', '/plone/Members/test_user_1_'),
('text', ''),
('title', ''),
('type', 'News Item')],
[('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain', 'simple_publication_workflow'),
('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state', 'private'),
('ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title', 'Document 4'),
('description', ''),
('id', 'doc4'),
('parent', ''),
('startpoint', '/plone/Members/test_user_1_'),
('text', ''),
('title', 'Document 4'),
('type', 'Document')],
[('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain', 'simple_publication_workflow'),
('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state', 'private'),
('ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title', 'mytest'),
('description', ''),
('id', 'sub1'),
('parent', ''),
('startpoint', '/plone/Members/test_user_1_'),
('text', ''),
('title', 'mytest'),
('type', 'Folder')],
[('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain', 'simple_publication_workflow'),
('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state', 'private'),
('ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title', 'Document 1 du dossier 1'),
('description', ''),
('id', 'doc11'),
('parent', 'sub1'),
('startpoint', '/plone/Members/test_user_1_'),
('text', ''),
('title', 'Document 1 du dossier 1'),
('type', 'Document')],
[('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain', 'simple_publication_workflow'),
('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state', 'private'),
('ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title', 'news 1 du dossier 1'),
('description', ''),
('id', 'news21'),
('parent', 'sub1'),
('startpoint', '/plone/Members/test_user_1_'),
('text', ''),
('title', ''),
('type', 'News Item')],
[('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain', 'simple_publication_workflow'),
('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state', 'private'),
('ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title', 'Document 2 du dossier 1'),
('description', ''),
('id', 'doc31'),
('parent', 'sub1'),
('startpoint', '/plone/Members/test_user_1_'),
('text', ''),
('title', 'Document 2 du dossier 1'),
('type', 'Document')]]
Import
Then, now that we got a working export, what about importing it
>>> id = self.folder.invokeFactory('Folder' , id='fa' , title="tests import")
>>> id = self.folder.invokeFactory('Folder' , id='fb' , title="tests import")
>>> fa = self.folder.fa; fb = self.folder.fb
Import in CSVReplicata plain format, an entry per line without contextual type hinting
>>> CSV = StringIO("""\
... "startpoint";"replicata_export_date";"parent";"id";"type";"title";"description";"ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title";"text"
... "Start point";"Export Date";"Parent folder";"Identifier";"Content type";"Title";"label_...";"ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title";"label_body_text"
... "/plone/Members/test_user_1_";20090101010101;"";"doc1";"Document";"Document 1";"";"Document 1";""
... "/plone/Members/test_user_1_";20090101010101;"";"doc2";"Document";"Document 2";"";"Document 2";""
... "/plone/Members/test_user_1_";20090101010101;"";"news1";"News Item";"";"";"news 'super' 3";""
... "/plone/Members/test_user_1_";20090101010101;"";"doc4";"Document";"Document 4";"";"Document 4";""
... "/plone/Members/test_user_1_";20090101010101;"";"sub1";"Folder";"mytest";"";"mytest";""
... "/plone/Members/test_user_1_";20090101010101;"sub1";"doc11";"Document";"Document 1 du dossier 1";"";"Document 1 du dossier 1";""
... "/plone/Members/test_user_1_";20090101010101;"sub1";"news21";"News Item";"";"";"news 1 du dossier 1";""
... "/plone/Members/test_user_1_";20090101010101;"sub1";"doc31";"Document";"Document 2 du dossier 1";"";"Document 2 du dossier 1";""
... """)
>>> print Icsvreplicata(self.folder.fa).csvimport(CSV, datetimeformat='%d%m%Y', delimiter=";", stringdelimiter='"', plain_format=True)
p...
(8, 0, ..., [])
>>> self.folder.fa.objectIds()
['doc1', 'doc2', 'news1', 'doc4', 'sub1']
>>> self.folder.fa.sub1.objectIds()
['doc11', 'news21', 'doc31']
Import in CSVReplicata orignal format, an entry per line with contextual type hinting
>>> CSV = StringIO("""\
... "/plone/Members/test_user_1_";20100101010101
... "parent";"id";"type";"title";"description";"text"
... "Parent folder";"Identifier";"Content type";"Title";"label_...";"label_body_text"
... "";"doc1";"Document";"Document 1";"";""
... "";"doc2";"Document";"Document 2";"";""
... "parent";"id";"type"
... "Parent folder";"Identifier";"Content type"
... "";"news1";"News Item"
... "parent";"id";"type";"title";"description";"text"
... "Parent folder";"Identifier";"Content type";"Title";"label_...";"label_body_text"
... "";"doc4";"Document";"Document 4";"";""
... "parent";"id";"type";"title";"description"
... "Parent folder";"Identifier";"Content type";"Title";"label_..."
... "";"sub1";"Folder";"mytest";""
... "parent";"id";"type";"title";"description";"text"
... "Parent folder";"Identifier";"Content type";"Title";"label_...";"label_body_text"
... "sub1";"doc11";"Document";"Document 1 du dossier 1";"";""
... "parent";"id";"type"
... "Parent folder";"Identifier";"Content type"
... "sub1";"news21";"News Item"
... "parent";"id";"type";"title";"description";"text"
... "Parent folder";"Identifier";"Content type";"Title";"label_...";"label_body_text"
... "sub1";"doc31";"Document";"Document 2 du dossier 1";"";""
... """)
>>> print Icsvreplicata(self.folder.fb).csvimport(CSV, datetimeformat='%d%m%Y', delimiter=";", stringdelimiter='"', plain_format=False)
p...
(8, 0, ..., [])
>>> self.folder.fb.objectIds()
['doc1', 'doc2', 'news1', 'doc4', 'sub1']
>>> self.folder.fb.sub1.objectIds()
['doc11', 'news21', 'doc31']
Default handlers
Verify the default handlers provided for Archetypes fields:
>>> handlersDict = self.portal.portal_csvreplicatatool.getHandlers()
>>> handlers = handlersDict.keys();handlers.sort()
>>> pprint(handlers)
['Products.Archetypes.Field.BooleanField',
'Products.Archetypes.Field.DateTimeField',
'Products.Archetypes.Field.FileField',
'Products.Archetypes.Field.FloatField',
'Products.Archetypes.Field.ImageField',
'Products.Archetypes.Field.IntegerField',
'Products.Archetypes.Field.LinesField',
'Products.Archetypes.Field.ReferenceField',
'Products.Archetypes.Field.StringField',
'Products.Archetypes.Field.TextField',
'Products.AttachmentField.AttachmentField.AttachmentField',
'default_handler',
...]
>>> pprint([(h, handlersDict[h]['handler_class']) for h in handlers], width=130)
[('Products.Archetypes.Field.BooleanField', <Products.csvreplicata.handlers.base.CSVBoolean object at ...>),
('Products.Archetypes.Field.DateTimeField', <Products.csvreplicata.handlers.base.CSVDateTime object at ...>),
('Products.Archetypes.Field.FileField', <Products.csvreplicata.handlers.file.CSVFile object at ...>),
('Products.Archetypes.Field.FloatField', <Products.csvreplicata.handlers.base.CSVFloat object at ...>),
('Products.Archetypes.Field.ImageField', <Products.csvreplicata.handlers.file.CSVFile object at ...>),
('Products.Archetypes.Field.IntegerField', <Products.csvreplicata.handlers.base.CSVInteger object at ...>),
('Products.Archetypes.Field.LinesField', <Products.csvreplicata.handlers.base.CSVLines object at ...>),
('Products.Archetypes.Field.ReferenceField', <Products.csvreplicata.handlers.reference.CSVReference object at ...>),
('Products.Archetypes.Field.StringField', <Products.csvreplicata.handlers.base.CSVString object at ...>),
('Products.Archetypes.Field.TextField', <Products.csvreplicata.handlers.base.CSVText object at ...>),
('Products.AttachmentField.AttachmentField.AttachmentField', <Products.csvreplicata.handlers.file.CSVFile object at ...>),
('default_handler', <Products.csvreplicata.handlers.base.CSVdefault object at ...>),
(...
What happened with csvreplicata during import/export if MyField is not in tool’s handlers. replicator.py apllies default_handler on it:
{'default_handler':
{'handler_class': base.CSVdefault(),'file': False}}
The replicator exporter/downloader
Export in normal mode
here we export folders and documents
>>> from Products.csvreplicata.browser import manager
>>> self.portal.portal_csvreplicatatool.replicabletypes = \
... {'Document':['default'], 'Folder':['default'],
... 'News Item': ['default'], 'File': ['default'] }
>>> import re
>>> self.setRoles(['Manager'])
>>> id=self.folder.invokeFactory('Document' , id='doc1' , title="Document 1")
>>> id=self.folder.invokeFactory('Document' , id='doc2' , title="Document 2")
>>> params = {"datetimeformat": '%d/%m/%Y %H:%M:%S',
... "vocabularyvalue": "No",
... "encoding": "UTF-8",
... "delimiter": ";",
... "stringdelimiter": '"',
... "exportable_content_types": ["News Items", "Document", "Folder"],
... }
>>> req = make_request('/'.join(self.folder.getPhysicalPath())+'@@csvreplicata', **params)
>>> repl = manager.ReplicationManager(self.folder, req)
>>> from Products.csvreplicata import replicator
>>> print ''.join([a for a in repl.doExport()])
"/plone/Members/test_user_1_";...
"parent";"id";"type";"title";"description";"text";...
"Parent folder";"Identifier";"Content type";"Title";"label_summary";...
"";"doc1";"Document";"Document 1";"";...
"";"doc2";"Document";"Document 2";"";...
<BLANKLINE>
>>> items = list(req.response._headers.iteritems());items.sort();pprint(items)
[('content-disposition', ['attachment; filename=export.csv']),
('content-length', ['...']),
('content-type', ['text/csv;charset=UTF-8'])]
Export as zip when there are files out there and we want them
Now adding and exporting a file:
>>> params['exportfiles'] = 'Yes'
>>> params["exportable_content_types"].append('File')
>>> id=self.folder.invokeFactory('File', id='f1' , title="File 1")
>>> req = make_request('/'.join(self.folder.getPhysicalPath())+'@@csvreplicata', **params)
>>> f1 = self.folder.f1
>>> f1.getFile().data = 'foo'
>>> f1.setFilename('bar')
>>> repl = manager.ReplicationManager(self.folder, req)
>>> from StringIO import StringIO
>>> import zipfile
>>> content = StringIO([a for a in repl.doExport()][0])
>>> zip = zipfile.ZipFile(content)
>>> print zip.read('export.csv')
"/plone/Members/test_user_1_";...
"parent";"id";"type";"title";"description";"...
"Parent folder";"Identifier";"Content type";"Title";"label_summary";...
"";"doc1";"Document";"Document 1";...
"";"doc2";"Document";"Document 2";...
"parent";"id";"type";"title";"description";...
"Parent folder";"Identifier";"Content ...
"";"f1";"File";"File 1";"";...
<BLANKLINE>
>>> zip.printdir()
File Name...
bar...
export.csv...
>>> items = list(req.response._headers.iteritems());items.sort();pprint(items)
[('content-disposition', ['attachment; filename=export.zip']),
('content-length', ['...']),
('content-type', ['application/zip'])]
The File Stream Iterator
This object returns a generator to read our big files!:
>>> from Products.csvreplicata.browser.manager import FileStreamIterator, EphemeralStreamIterator
>>> import tempfile
>>> fpath = tempfile.mkstemp('foo')[1]
>>> fobj = open(fpath, 'w');fobj.write('foo');fobj.close()
>>> len(FileStreamIterator(fpath))
3
We can play with chunks to divide rendering into small parts
>>> [[a for a in FileStreamIterator(fpath, chunk=chunk)] for chunk in [1,2, 3,4]] [['f', 'o', 'o'], ['fo', 'o'], ['foo'], ['foo']] >>> os.unlink(fpath)
The Ephemeral Stream Iterator
This object returns a generator to read our big files but delete them when they are read!:
>>> from Products.csvreplicata.browser.manager import FileStreamIterator, EphemeralStreamIterator
>>> import tempfile
>>> fdir = tempfile.mkdtemp(); fdir2 = tempfile.mkdtemp(dir=fdir);fpath = os.path.join(fdir2, 'foo')
>>> fobj = open(fpath, 'w');fobj.write('foo');fobj.close()
Files are there, we can ask to not delete parents (default):
>>> [os.path.exists(p) for p in fdir,fdir2, fpath] [True, True, True] >>> len(EphemeralStreamIterator(fpath, delete_parent=False, delete_grand_parent=False)) 3 >>> [a for a in EphemeralStreamIterator(fpath, delete_parent=False, delete_grand_parent=False)] ['foo'] >>> [os.path.exists(p) for p in fdir,fdir2, fpath] [True, True, False]
We have read it, the file and the parent are deleted:
>>> fobj = open(fpath, 'w');fobj.write('foo');fobj.close()
>>> [a for a in EphemeralStreamIterator(fpath, delete_parent=True, delete_grand_parent=True)]
['foo']
>>> [os.path.exists(p) for p in fdir2, fdir, fpath]
[False, False, False]
Changelog
1.1.9 (2012-05-28)
Nothing changed yet.
1.1.8 (2012-05-28)
fix blob files export/import [kiorky]
handle years < 999 [kiorky]
let commits be more incremental [kiorky]
fix export order [kiorky]
improve retry policy [kiorky]
1.1.7 (2010-08-20)
Adding comments exporter/searcher [kiorky]
1.1.6 (2010-07-30)
Fix importObject (obj referenced before assignment) [danjacka]
1.1.5 (2010-07-29)
Fix unittest for plone4 msgid changes [kiorky]
Implement import plugins [kiorky]
Add a meta.zcml with some basic plugins, like for wf state export/import [kiorky]
Rename objects for consistency purpose [kiorky]
Fix some Acquisition bugs [kiorky]
Support blobstorage [kiorky]
1.1.4 (2010-03-05)
fix some encoding issues with file handler [kiorky]
better plain format import [kiorky]
Fix setSubject [kiorky]
1.1.3 (2010-01-27)
Fixed missing import of interface.implements in browser.manager [fRiSi]
1.1.2 (2010-01-27)
Fix silly empty list bug in replicator.csvimport prototype. [kiorky]
fixed missing import of interface.implements in browser.manager [fRiSi]
Fix unicode error with reference fields [kiorky]
Add support for temporary export path [kiorky]
Add support for flattened CSV Files [kiorky]
Add tests, and tests infrastructure [kiorky]
Add Stream (Files) Iterators not to overhead the RAM [kiorky]
1.1.1 - Unreleased
remove five:traversable directive (deprecated in plone3) [toutpt]
Add default config for importcsvStep [toutpt]
importcsvStep now depend on plone-final step [toutpt]
option ignore_content_errors added to import step config (set it to true) and to the replicator methods. It allows to log errors when setting fields instead of raising exceptions and stop. [kiorky]
1.1 (2009-10-17)
A new import step to use csvreplicata to import contents [toutpt]
Fix creationflag issue [toutpt]
plugins system. Export other things than AT. take a look at interfaces and adpaters [kiorky]
some encoding bugs fixed [kiorky]
datetime format settings in the tool [kiorky]
images/files created with folder structure in export [kiorky]
Plone 2.5 compatibility back [kiorky]
1.0.7 (2009-07-15)
Prevent fixTools from running outside of the context of Products.csvreplicata [pigeonflight]
Fix export when data value is not ascii [yboussard]
1.0.6
support vocabulary defined as object method [Jim BAACK]
1.0.5
fix bug crc32 with large media files
when you set a CSVHandledTypesSchematas default is automatically preselected for user convenience [Jean-Philippe CAMGUILHEM]
1.0.4
Uninstall problem fix
IMPORTANT NOTE: when migrating from previous versions, you need to uninstall csvreplicata and then reinstall csvreplicata. [Jean-Philippe CAMGUILHEM]
1.0.3
Custom handlers can be now implemented outside the product, and dynamically declared to the csvreplicata tool. [Jean-Philippe CAMGUILHEM]
1.0.2
Initial release.[Eric BREHAULT / Christophe SAUTHIER]
1.0
Unreleased
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Products.csvreplicata-1.1.9.zip
(90.9 kB
view hashes)
