plone.app.blob 1.8.1

Overview

This package aims to be an add-on for Plone (>= 3.x) integrating ZODB (>=3.8) blob support, which allows large binary data to be managed by the ZODB, but separately from your usual FileStorage database, i.e. Data.fs. This has several advantages, most importantly a much smaller Data.fs and better performance both cpu- as well as memory-wise.

Contents

Status

At the moment the integration for “File” content should be stable, but still needs more field testing. It is being successfully used in several production deployments, though. The provided blob-based content type should safely usable as a drop-in replacement for ATFile. As such it has been successfully tested against all CMFPlone and ATContentTypes tests. Please use the provided test-compatibility.sh script to run these tests for yourself.

Image support is still in an alpha stadium and not enabled by default. It can be activated by applying the respective profile via the portal setup tool.

More detailed information about the integration and the current status can be found in the corresponding Plone enhancement and Plone 4 PLIP tickets.

Requirements

Plone 3.0 or newer is required. The package has been tested with all versions from 3.0 up to and including 4.0. However, as all versions before 3.0.4 require a workaround described in the Troubleshooting section below, it is recommended to use Plone 3.0.4 or a more recent version.

Installation

The easiest way to get ZODB blob support in Plone 3 using this package is to work with installations based on zc.buildout. Other types of installations should also be possible, but might turn out to be somewhat tricky — please see the FAQ section below.

To get started you will simply need to add the package to your “eggs” and “zcml” sections, run buildout, restart your Plone instance and install the “plone.app.blob” package using the quick-installer or via the “Add-on Products” section in “Site Setup”.

A sample buildout configuration file, i.e. buildout.cfg, could look like this:

[buildout]
parts = zope2 instance
extends = http://dist.plone.org/release/3.3.1/versions.cfg
find-links =
    http://dist.plone.org/release/3.3.1
    http://dist.plone.org/thirdparty/
versions = versions

[versions]
ZODB3 = 3.8.3

[zope2]
recipe = plone.recipe.zope2install
url = ${versions:zope2-url}

[instance]
recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
blob-storage = var/blobstorage
user = admin:admin
eggs =
    Plone
    plone.app.blob
zcml = plone.app.blob

You can also use this buildout configuration to create a fresh Plone installation. To do so you would store it as buildout.cfg — preferably in an empty directory, download bootstrap.py into the same directory and issue the following commands:

$ python bootstrap.py
$ ./bin/buildout
$ ./bin/instance fg

After that you create a “Plone Site” via the ZMI as usual and either select the “plone.app.blob” extension profile at creation time or again install the “plone.app.blob” package using one of the above mentioned methods.

A sample ZEO buildout configuration could look like this:

[buildout]
parts = zope2 zeoserver instance1 instance2
extends = http://dist.plone.org/release/3.3.1/versions.cfg
find-links =
    http://dist.plone.org/release/3.3.1
    http://dist.plone.org/thirdparty/
versions = versions

[versions]
ZODB3 = 3.8.3

[zope2]
recipe = plone.recipe.zope2install
url = ${versions:zope2-url}

[zeoserver]
recipe = plone.recipe.zope2zeoserver
zope2-location = ${zope2:location}
zeo-address = 127.0.0.1:8100
zeo-var = ${buildout:directory}/var
blob-storage = ${zeoserver:zeo-var}/blobstorage
eggs = plone.app.blob

[instance1]
recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
zeo-address = ${zeoserver:zeo-address}
blob-storage = ${zeoserver:blob-storage}
zeo-client = on
shared-blob = on
user = admin:admin
eggs =
    Plone
    plone.app.blob
zcml = plone.app.blob

[instance2]
recipe = plone.recipe.zope2instance
http-address = 8081
zope2-location = ${instance1:zope2-location}
zeo-client = ${instance1:zeo-client}
zeo-address = ${instance1:zeo-address}
blob-storage = ${instance1:blob-storage}
shared-blob = ${instance1:shared-blob}
user = ${instance1:user}
eggs = ${instance1:eggs}
zcml = ${instance1:zcml}

Please note the configuration options blob-storage and shared-blob specified in [client1] and [client2]. To enable blob support on a ZEO client (or standalone instance) you always have to specify a path in the blob-storage configuration option. If shared-blob is set to “on”, the ZEO client will assume it can read blob files directly from within the path specified in the blob-storage option. This path might also refer to a network share in case the ZEO client and server are installed on separate machines. However, to stream blob files trough the ZEO connection you will have to set the shared-blob option to “off”. The path specified in the blob-storage option will be ignored in this situation, but it needs to be set nevertheless.

More detailed instructions on how to set things up as well as some background information on blobs — or in other words the story of an “early adopter” — can be found in Ken Manheimer’s wiki. This is a highly useful resource and recommended read for people trying to give blobs a spin. Please note however, that most of the recipe changes described in these instructions have already been incorporated in the particular recipes by now.

In addition, more information on how to use buildout is available in the accompanying README.txt as well as in Martin’s excellent buildout tutorial on plone.org.

Migrating existing content

In-place content migration is provided for existing “File” and “Image” content. The Products.contentmigration package is required for this to work. To install this package you will again need to add its name to the “eggs” and “zcml” section of your buildout.cfg, so that it reads like:

[instance]
...
eggs +=
    plone.app.blob
    Products.contentmigration
zcml +=
    plone.app.blob
    Products.contentmigration

You can also refer to the above mentioned sample buildout.cfg for details.

In order to then migrate your existing file content to blobs you can use the migration interfaces provided at http://<site>/@@blob-file-migration to migrate “File” content as well as http://<site>/@@blob-image-migration for “Image” content respectively. <site> will need to be replaced with the URL of your “Plone Site” object here, of course. The pages will show you the number of available ATFile or ATImage instances and then lets you convert these to the provided blob content types by clicking a button.

For custom AT-based content types that use FileField(s), see example.blobattype for details of how to enable and migrate them to use blobs.

Please refer to the next section if you encounter any errors during migration.

Troubleshooting

The following are some known issues, that will hopefully be resolved soon enough. In the meantime here are the recommended workarounds:

“AttributeError: ‘module’ object has no attribute ‘VersionBase’” Exception

Symptom

After upgrading your buildout you’re getting errors like the following:

Traceback (innermost last):
  ...
  Module App.PersistentExtra, line 57, in locked_in_version
AttributeError: 'module' object has no attribute 'VersionBase'

Problem

Version 1.0b5 of plone.app.blob adds support for Plone 4 as well as Dexterity, which is why the version restriction for ZODB had to be lifted. However, while Plone 4 will use Zope 2.12 and ZODB 3.9, Plone 3.x doesn’t work with either of these.

Solution

Downgrade ZODB3 to a release from the 3.8 series. You can do this by adding a version pin like:

[versions]
ZODB3 = 3.8.3

to your buildout.cfg.

“FileFieldException: Value is not File or String (…)” Exception

Symptom

After upgrading your buildout you’re getting an error like the following during blob migration:

Traceback (innermost last):
  File ".../basemigrator/walker.py", line 174, in migrate
  ...
  File ".../Archetypes/Field.py", line 931, in _process_input
FileFieldException: Value is not File or String (...)

Problem

Your version of archetypes.schemaextender has been upgraded to 1.1 while running buildout. You either didn’t run it in non-newest mode (-N) or have not pinned down the version of archetypes.schemaextender.

Solution

Downgrade archetypes.schemaextender to version 1.0 for the moment. You can do this by adding a version pin like:

[versions]
archetypes.schemaextender = 1.0

to your buildout.cfg. A proper fix to add compatibility to the latest version is being worked on.

“AttributeError: ‘NoneType’ object has no attribute ‘getAccessor’” Exception

Symptom

After upgrading from version 1.0b2 or earlier you’re getting an error like the following when trying to view blob-based content:

Traceback (innermost last):
  Module ZPublisher.Publish, line 119, in publish
  ...
  Module Products.ATContentTypes.content.base, line 300, in get_content_type
AttributeError: 'NoneType' object has no attribute 'getAccessor'

Problem

Recent versions have added support for sub-types based on marker interfaces and your existing blob-based content hasn’t been marked yet.

Solution

Upgrade to at least 1.0b4, re-install “plone.app.blob” via the quick-installer and reset all sub-types by accessing the @@blob-maintenance/resetSubtypes view.

“Invalid plugin id” Exception

Symptom

When trying to create a “Plone Site” you’re getting an error like:

Error Type: KeyError
Error Value: 'Invalid plugin id: credentials_basic_auth'

Problem

Your version of Products.PluggableAuthService is too old — you need 1.5.2 or newer (please see http://www.zope.org/Collectors/PAS/59 for more information about this).

Solution

Please use the provided buildout, add the 1.5 branch as an svn:external to the products/ directory of your buildout or upgrade to Plone 3.0.4 by re-running buildout.

“unknown type name: ‘blobstorage’”

Symptom

When running buildout you’re getting an error like:

Error: unknown type name: 'blobstorage'
(line 36 in file:///.../parts/instance/etc/zope.conf)

Problem

Your version of the plone.recipe.zope2instance recipe is too old — you need to have at least version 1.0.

Solution

Make sure you’re running buildout with neither “-N” nor “-o” and you also don’t have:

newest = false

in your ~/.buildout/default.cfg. Alternatively, running buildout with option “-n” should update the recipe to the latest version.

missing distribution for required “zdaemon” and “ZConfig” eggs

Symptom

When running buildout you’re getting errors like:

Getting distribution for 'zdaemon>=1.4a2,<1.4.999'.
While:
  Installing instance.
  Getting distribution for 'zdaemon>=1.4a2,<1.4.999'.
Error: Couldn't find a distribution for 'zdaemon>=1.4a2,<1.4.999'.

or:

Getting distribution for 'ZConfig>=2.4a2,<2.4.999'.
While:
  Installing instance.
  Getting distribution for 'ZConfig>=2.4a2,<2.4.999'.
Error: Couldn't find a distribution for 'ZConfig>=2.4a2,<2.4.999'.

Problem

zdaemon and ZConfig eggs have only been released to the Cheeseshop starting from more recent versions, i.e. 2.0 and 2.5 respectively. Older distributions in egg format are only available from http://download.zope.org/distribution

Solution

Add the above link to the find-links setting of the [buildout] section in your buildout.cfg, like:

find-links =
    http://download.zope.org/distribution/
    ...

“ZRPCError: bad handshake ‘Z303’”

Symptom

With a ZEO setup you are getting errors like:

ZRPCError: bad handshake 'Z303'

Problem

You probably haven’t added plone.app.blob to the eggs setting in your [zeo] buildout part. Without it the ZEO server will not use the required version 3.8 of ZODB and hence not support blobs.

Solution

Add the string plone.app.blob to the eggs setting in the [zeo] section (i.e. the one using the plone.recipe.zope2zeoserver recipe) in your buildout.cfg, like:

[zeo]
...
eggs = plone.app.blob
...

“AttributeError: ‘NoneType’ object has no attribute ‘product’” during migration

Symptom

After installing “plone.app.blob” via the quick-installer or applying the “plone.app.blob: ATFile replacement” profile you are seeing migration errors like:

Traceback (innermost last):
  Module ZPublisher.Publish, line 119, in publish
  Module ZPublisher.mapply, line 88, in mapply
  Module ZPublisher.Publish, line 42, in call_object
  Module plone.app.blob.browser.migration, line 24, in __call__
  Module plone.app.blob.migrations, line 42, in migrateATFiles
  Module Products.contentmigration.basemigrator.walker, line 126, in go
  Module Products.contentmigration.basemigrator.walker, line 205, in migrate
MigrationError: MigrationError for obj at /... (File -> Blob):
Traceback (most recent call last):
  File ".../Products/contentmigration/basemigrator/walker.py", line 174, in migrate
    migrator.migrate()
  File ".../Products/contentmigration/basemigrator/migrator.py", line 185, in migrate
    method()
  File ".../Products/contentmigration/archetypes.py", line 111, in beforeChange_schema
    archetype = getType(self.dst_meta_type, fti.product)
AttributeError: 'NoneType' object has no attribute 'product'

Problem

The current migration code has been written to convert existing “File” content to the “Blob” content type provided by the base “plone.app.blob” profile. However, that type isn’t known when just installing the “ATFile replacement” profile. The latter is probably what you want to install, though, as former “File” content will keep the same portal type, i.e. “File” after being migrated. This way no apparent changes are visible, which might help with avoiding confusion.

Solution

For now you might work around this by either applying the “plone.app.blob” profile via the ZMI in /portal_setup. This will install the above mentioned “Blob” content type. After that migration will work, but your former “File” content will have the “Blob” content type.

If that’s not what you want, simply change line line 17 in plone/app/blob/migrations.py (which is probably contained in an egg directory located somewhere like eggs/plone.app.blob-1.0b2-py2.4.egg/ relative to your buildout/installation) from:

dst_portal_type = 'Blob'

to:

dst_portal_type = 'File'

After that migration should use the new “File” type, based on ZODB blobs. Once you’ve migrated you might remove or disable the “Blob” type from /portal_types again. A future version of “plone.app.blob” will try auto-detect the correct target type for the migration (or at least allow to specify it) to make this more convenient.

If you have already migrated to “Blob” content, but would rather like to have “File” items, you can change the two previous lines to:

src_portal_type = 'Blob'
src_meta_type = 'ATBlob'

and re-run the blob migration. This will convert your “Blob”s to show up as “File”s again. You should probably pack your ZODB afterwards to avoid having its blob storage occupy twice as much disk space as actually needed (the extra migration will create new blobs).

“Image” and/or “File” content doesn’t show up as expected after migrating to blobs

Symptom

After migrating “Image” and/or “File” content to be based on blobs, some of it doesn’t show up as expected. A typical example of this are ATCT’s photo album views.

Problem

All versions before 1.0b11 didn’t update the “Type” catalog index correctly during migration. This could of course result in wrong results for all queries using this index.

Solution

Manually update the “Type” index using the ZMI or upgrade to at least 1.0b11 and use the @@blob-maintenance/updateTypeIndex view to limit the reindexing to only blob-based content. The latter should usually be quicker, especially for bigger sites.

Errors when using additionally mounted databases

Symptom

With additionally configured ZODB mount-points you are getting errors like:

Traceback (innermost last):
  ...
  Module ZEO.ClientStorage, line 1061, in temporaryDirectory
AttributeError: 'NoneType' object has no attribute 'temp_dir

or:

Traceback (innermost last):
  ...
  Module ZODB.blob, line 495, in temp_dir
TypeError: Blobs are not supported

Problem

You haven’t configured a blob-storage for your extra database.

Solution

Please refer to David Glick’s comment in ticket #10130 for detailed information about the various ways to configure a blob-storage for additional mount-points. The recommended way to accomplish this both for ZEO and non-ZEO setups is to use collective.recipe.filestorage and adjust your buildout with the following:

[buildout]
...
parts =
    ...
    filestorage
    instance

[filestorage]
recipe = collective.recipe.filestorage
blob-storage = var/blobstorage-%(fs_part_name)s
parts =
    foo

Please note that for the “parts” setting in the “buildout” section it is important to list “filestorage” before any parts installing Zope or ZEO. The “parts” setting in the “filestorage” section, however, represents a list of filestorage sub-parts to be generated, one per line. Further details can be found in the documentation of the recipe.

FAQ

Is it possible to use “plone.app.blob” in installations not based on zc.buildout?

Yes, but that would require some additional steps, since it depends on ZODB 3.8, but Plone currently ships with Zope 2.10, which still comes with ZODB 3.7. So, to make things work you could either install the required versions of all additionally needed packages into your lib/python/ directory or use the respective eggs and make sure they get preferred over their older versions on import, for example by setting up PYTHONPATH.

Alternatively it should also be possible to install the package using easy_install, which would automatically install its dependencies including ZODB 3.8, too. Again you would need to set up your PYTHONPATH to make sure the desired versions are used. However, installing the package like this is likely to have side effects on other Zope/Plone instances on your system, so you probably want to use virtualenv here at least.

Overall, to get started without too much pain, a buildout-based installation is recommended — for example the provided buildout.

Will this be available for Plone 2.5.x?

Yes, support for the 2.5 series is planned and next on the agenda.

What about image support, i.e. a drop-in for ATImage content?

While just replacing the primary field in ATImage’s schemata should probably already work quite well, proper image support is planned for a later release. “proper” here means using a sub-typing approach as presented by Rocky Burt in Naples, which will have several advantages including a cleaner and better structured code, but will also take a little longer to implement.

Strange messages like Exception exceptions.OSError: (2, 'No such file or directory', '.../tmpZvxjZB') in <bound method _TemporaryFileWrapper.__del__ of <closed file '<fdopen>', mode 'w+b' at 0x7317650>> ignored get written to the logs whenever a file is uploaded. Is that an error or something to worry about?

No, that’s fine, it’s just a small annoyance, that should be fixed eventually. In case you care, the problem is that the zope publisher creates a temporary file for each upload it receives. Once the upload has finished that temporary file is passed to the blob machinery, which moves it into its blob storage. However, at the end of the request the wrapper class for temporary files tries to remove the file as well, since well, it’s supposed to be temporary. At that time the file is already gone though, and the above warning is issued.

I have a ZEO setup with the server and clients running on separate machines. Why do I get blobs stored in my ZEO clients’ blobstorage directories and not only on the server?

ZEO clients cache blobs the first time they are fetched. Unfortunately the cache is not cleaned automatically when the instances are stopped and will keep growing. In addition, if you manually delete the files without restarting, the ZEO client will still expect to find them. ZODB 3.9, which is used by Plone 4, introduces a cache size control that alleviates the problem. Plone 3.x and earlier can only be used with ZODB 3.8.x, though. However, Sasha Vincic has written a workaround for Plone 2.5.x that invalidates the existing reference causing the blob data to be fetched again from the ZEO server should it be missing. The patch has been merged and is available from version 1.0b11.

Feedback

Any kind of feedback like bug reports, suggestions, feature requests and most preferably success stories is most welcome and much appreciated. Especially, it would be interesting to hear about success or problems with migration of existing content and installations on platforms other than OSX.

So please feel free to file tickets in the issue tracker, contact me on #plone, #plone-framework, the plone developer mailing list or directly via email.

Detailed Documentation

This package integrates ZODB 3.8’s blob support into Plone 3.0. To do this a new content type Blob is provided, which can be used instead of the existing File and Image types. Their behaviour is mimicked by sub-typing, which in this case means dynamically changing views and schema of the underlying Blob type as well as adapting it to add functionality.

First of all the plone.app.blob package needs to be installed, which at the moment requires a special branch of Zope 2.10 as well as a few additional packages for extending the schema and migration purposes. The easiest way to get a working setup is probably to use one of the provided buildout configurations, either one based on ploneout and therefore mainly targeted at developers or another based on plone.recipe.plone. The latter uses the current plone release tarball instead of subversion checkout, meaning it is mainly targeted at integrators and users (and significantly faster to set up as well :)).

In any way, the setup should make the new content type available as well as instantiable:

>>> from Products.CMFCore.utils import getToolByName
>>> portal = layer['portal']
>>> portal_types = getToolByName(portal, 'portal_types')
>>> portal_types.getTypeInfo('Blob')
<DynamicViewTypeInformation at /plone/portal_types/Blob>

>>> from plone.app.testing import TEST_USER_ID
>>> folder = portal.portal_membership.getHomeFolder(TEST_USER_ID)
>>> folder.invokeFactory('Blob', id='blob', title='a Blob')
'blob'
>>> blob = folder.blob
>>> blob
<ATBlob at /plone/Members/test_user_1_/blob>

The new instance should have been marked with the default sub-type and therefore also contain the extended schema:

>>> from plone.app.blob.interfaces import IATBlobBlob
>>> IATBlobBlob.providedBy(blob)
True
>>> blob.getField('file')
<Field file(blob:rw)>

Mimicking the existing “File” content type, i.e. ATFile, it shouldn’t have an associated workflow:

>>> workflow_tool = getToolByName(portal, 'portal_workflow')
>>> workflow_tool.getWorkflowsFor(blob)
[]

Since no data has been written to it, the blob file should still be empty:

>>> blob.getFile().getBlob()
<ZODB.blob.Blob object at ...>
>>> blob.getFile().getBlob().open().read()
''

Feeding it with some image data should result in a correctly set mime-type and a now non-empty blob file:

>>> from StringIO import StringIO
>>> from base64 import decodestring
>>> gif = 'R0lGODlhAQABAPAAAPj8+AAAACH5BAEAAAAALAAAAAABAAEAAAICRAEAOw=='
>>> gif = StringIO(decodestring(gif))
>>> blob.setFile(gif)
>>> print blob.getFilename()
None
>>> blob.getContentType()
'image/gif'
>>> len(blob.getFile().getBlob().open().read())
43
>>> str(blob) == gif.getvalue()
True

Migration from existing file content, i.e. ATFile instances, is also provided. The payload data as well as all other fields should be properly migrated:

>>> initial_file_product = portal.portal_types.File.product
>>> initial_file_factory = portal.portal_types.File.factory
>>> portal.portal_types.File.product = 'ATContentTypes'
>>> portal.portal_types.File.factory = 'addATFile'
>>> gif.filename = 'foo.gif'
>>> folder.invokeFactory('File', id='foo', title='a file', file=gif,
...     subject=('foo', 'bar'), contributors=('me'))
'foo'
>>> portal.portal_types.File.product = initial_file_product
>>> portal.portal_types.File.factory = initial_file_factory
>>> folder.foo
<ATFile at /plone/Members/test_user_1_/foo>
>>> folder.foo.Title()
'a file'
>>> folder.foo.getFilename()
'foo.gif'
>>> folder.foo.getContentType()
'image/gif'
>>> folder.foo.Subject()
('foo', 'bar')
>>> folder.foo.Contributors()
('me',)

>>> from plone.app.blob.migrations import migrateATFiles
>>> migrateATFiles(portal)
'Migrating /plone/Members/test_user_1_/foo (File -> Blob)\n'

>>> folder.foo
<ATBlob at /plone/Members/test_user_1_/foo>
>>> folder.foo.Title()
'a file'
>>> folder.foo.getFilename()
'foo.gif'
>>> folder.foo.getContentType()
'image/gif'
>>> folder.foo.Subject()
('foo', 'bar')
>>> folder.foo.Contributors()
('me',)
>>> folder.foo.getFile().getBlob()
<ZODB.blob.Blob object at ...>
>>> str(folder.foo) == gif.getvalue()
True
>>> folder.foo.getFile().getBlob().open().read()
'GIF89a...'

Also, migrating should have indexed the new content correctly to prevent stale or wrong data from showing up in some views, i.e. folder listing:

>>> catalog = getToolByName(portal, 'portal_catalog')
>>> brain = catalog(id = 'foo')[0]
>>> folder.foo.UID() == brain.UID
True

>>> folder.foo.getObjSize() == brain.getObjSize
True

Finally the correct creation of blob-based content “through the web” is tested using a testbrowser:

>>> from plone.app.testing import setRoles
>>> setRoles(portal, TEST_USER_ID, ['Editor'])

>>> from plone.testing.z2 import Browser

>>> from plone.app.testing import TEST_USER_NAME, TEST_USER_PASSWORD
>>> browser = Browser(layer['app'])
>>> browser.addHeader('Authorization', 'Basic %s:%s' % (
...     TEST_USER_NAME, TEST_USER_PASSWORD))

>>> browser.open(folder.absolute_url())
>>> browser.getLink(url='createObject?type_name=Blob').click()
>>> browser.url
'http://nohost/plone/.../portal_factory/Blob/blob.../edit...'
>>> browser.getControl(name='title').value = 'Foo bar'
>>> control = browser.getControl(name='file_file')
>>> testfile = StringIO('%PDF-1.4 fake pdf...' + 'foo' * 1000)
>>> control.add_file(testfile, None, 'foo.pdf')
>>> browser.getControl('Save').click()
>>> browser.url
'http://nohost/plone/.../foo-bar/view'
>>> browser.contents
'...Info...Changes saved...
 ...Foo bar...foo.pdf...PDF document...'

Changelog