plone.app.blob 1.0b2

Overview

This package aims to be an add-on for Plone 3.x integrating ZODB 3.8’s blob support, which allows large binary data to be managed by the ZODB, but separately from your usual FileStorage database, i.e. Data.fs. This has several advantages, most importantly a much smaller Data.fs and better performance both cpu- as well as memory-wise.

Status

At the moment the integration is in a working state, but needs more field testing. The provided blob-based content type should be more or less safely usable as a drop-in replacement for ATFile. As such it has been successfully tested against all CMFPlone and ATContentTypes tests. Instructions for replicating the necessary test setup and running these tests yourself can be found at http://dev.plone.org/plone/changeset/18321.

More detailed an up-to-date information about the integration and the current status can be found in the corresponding plone enhancement ticket.

Requirements

Plone 3.0 or newer is required. The package has been tested with all versions from 3.0 up to and including 3.0.4. However, as all versions before 3.0.4 require a workaround described in the Troubleshooting section below, it is recommended to use Plone 3.0.4.

Installation

The easiest way to get ZODB blob support in Plone 3 using this package is to work with installations based on zc.buildout. Other types of installations should also be possible, but might turn out to be somewhat tricky — please see the FAQ section below.

To get started you will simply need to add the package to your “eggs” and “zcml” sections, run buildout, restart your Plone instance and install the “plone.app.blob” package using the quick-installer or via the “Add-on Products” section in “Site Setup”.

A sample buildout configuration file, i.e. buildout.cfg, could look like this:

[buildout]
parts = plone zope2 instance
find-links =
    http://dist.plone.org
    http://download.zope.org/ppix/
    http://download.zope.org/distribution/
    http://effbot.org/downloads
eggs = elementtree

[plone]
recipe = plone.recipe.plone

[zope2]
recipe = plone.recipe.zope2install
url = ${plone:zope2-url}

[instance]
recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
blob-storage = var/blobstorage
user = admin:admin
products = ${plone:products}
eggs =
    ${buildout:eggs}
    ${plone:eggs}
    plone.app.blob
zcml = plone.app.blob

You can also use this buildout configuration to create a fresh Plone installation. To do so you would store it as buildout.cfg — preferably in an empty directory, download bootstrap.py into the same directory and issue the following commands:

$ python bootstrap.py
$ ./bin/buildout
$ ./bin/instance fg

After that you create a “Plone Site” via the ZMI as usual and either select the “plone.app.blob” extension profile at creation time or again install the “plone.app.blob” package using one of the above mentioned methods.

For your convenience a working buildout configuration, including bootstrap.py and buildout.cfg, is provided as a subversion checkout at http://svn.plone.org/svn/plone/plone.app.blob/buildouts/plone-3.0.

A sample ZEO buildout configuration could look like this:

[buildout]
parts = plone zope2 zeo client1 client2
find-links =
  http://dist.plone.org
  http://download.zope.org/ppix/
  http://download.zope.org/distribution/
  http://effbot.org/downloads
eggs = elementtree

[plone]
recipe = plone.recipe.plone

[zope2]
recipe = plone.recipe.zope2install
url = ${plone:zope2-url}

[zeo]
recipe = plone.recipe.zope2zeoserver
zope2-location = ${zope2:location}
zeo-address = 127.0.0.1:8100
zeo-var = ${buildout:directory}/var
blob-storage = ${zeo:zeo-var}/blobstorage
eggs = plone.app.blob

[client1]
recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
zeo-address = ${zeo:zeo-address}
blob-storage = ${zeo:blob-storage}
zeo-client = on
shared-blob = on
user = admin:admin
products = ${plone:products}
eggs =
  ${buildout:eggs}
  ${plone:eggs}
  plone.app.blob
zcml = plone.app.blob

[client2]
recipe = plone.recipe.zope2instance
http-address = 8081
zope2-location = ${client1:zope2-location}
zeo-client = ${client1:zeo-client}
zeo-address = ${client1:zeo-address}
blob-storage = ${client1:blob-storage}
shared-blob = ${client1:shared-blob}
user = ${client1:user}
products = ${client1:products}
eggs = ${client1:eggs}
zcml = ${client1:zcml}

More detailed instructions on how to set things up as well as some background information on blobs — or in other words the story of an “early adopter” — can be found in Ken Manheimer’s wiki. This is a highly useful resource and recommended read for people trying to give blobs a spin. Please note however, that most of the recipe changes described in these instructions have already been incorporated in the particular recipes by now.

In addition, more information on how to use buildout is available in the accompanying README.txt as well as in Martin’s excellent buildout tutorial on plone.org.

Migrating existing content

In-place content migration is provided for existing ATFile content. The Products.contentmigration package is required for this to work. To install this package you will again need to add its name to the “eggs” and “zcml” section of your buildout.cfg, so that it reads like:

[instance]
...
eggs =
    ${buildout:eggs}
    ${plone:eggs}
    plone.app.blob
    Products.contentmigration
zcml =
    plone.app.blob
    Products.contentmigration

You can also refer to the above mentioned sample buildout.cfg for details.

In order to then migrate your existing file content to blobs you can use the migration interface provided at http://localhost:8080/plone/@@blob-migration, where “plone” should be replaced with the id of your “Plone Site” object. The page will show you the number of available ATFile instances and lets you convert them to the provided blob content type by clicking a button.

Troubleshooting

The following are some known issues, that will hopefully be resolved soon enough. In the meantime here are the recommended workarounds:

“Invalid plugin id” Exception

Symptom

When trying to create a “Plone Site” you’re getting an error like:

Error Type: KeyError
Error Value: 'Invalid plugin id: credentials_basic_auth'

Problem

Your version of Products.PluggableAuthService is too old — you need 1.5.2 or newer (please see http://www.zope.org/Collectors/PAS/59 for more information about this).

Solution

Please use the provided buildout, add the 1.5 branch as an svn:external to the products/ directory of your buildout or upgrade to Plone 3.0.4 by re-running buildout.

“unknown type name: ‘blobstorage’”

Symptom

When running buildout you’re getting an error like:

Error: unknown type name: 'blobstorage'
(line 36 in file:///.../parts/instance/etc/zope.conf)

Problem

Your version of the plone.recipe.zope2instance recipe is too old — you need to have at least version 1.0.

Solution

Make sure you’re running buildout with neither “-N” nor “-o” and you also don’t have:

newest = false

in your ~/.buildout/default.cfg. Alternatively, running buildout with option “-n” should update the recipe to the latest version.

missing distribution for required “zdaemon” and “ZConfig” eggs

Symptom

When running buildout you’re getting errors like:

Getting distribution for 'zdaemon>=1.4a2,<1.4.999'.
While:
  Installing instance.
  Getting distribution for 'zdaemon>=1.4a2,<1.4.999'.
Error: Couldn't find a distribution for 'zdaemon>=1.4a2,<1.4.999'.

or:

Getting distribution for 'ZConfig>=2.4a2,<2.4.999'.
While:
  Installing instance.
  Getting distribution for 'ZConfig>=2.4a2,<2.4.999'.
Error: Couldn't find a distribution for 'ZConfig>=2.4a2,<2.4.999'.

Problem

zdaemon and ZConfig eggs have only been released to the Cheeseshop starting from more recent versions, i.e. 2.0 and 2.5 respectively. Older distributions in egg format are only available from http://download.zope.org/distribution

Solution

Add the above link to the find-links setting of the [buildout] section in your buildout.cfg, like:

find-links =
    http://download.zope.org/distribution/
    ...

“ZRPCError: bad handshake ‘Z303’”

Symptom

With a ZEO setup you are getting errors like:

ZRPCError: bad handshake 'Z303'

Problem

You probably haven’t added plone.app.blob to the eggs setting in your [zeo] buildout part. Without it the ZEO server will not use the required version 3.8 of ZODB and hence not support blobs.

Solution

Add the string plone.app.blob to the eggs setting in the [zeo] section (i.e. the one using the plone.recipe.zope2zeoserver recipe) in your buildout.cfg, like:

[zeo]
...
eggs = plone.app.blob
...

FAQ

Is it possible to use “plone.app.blob” in installations not based on zc.buildout?

Yes, but that would require some additional steps, since it depends on ZODB 3.8, but Plone currently ships with Zope 2.10, which still comes with ZODB 3.7. So, to make things work you could either install the required versions of all additionally needed packages into your lib/python/ directory or use the respective eggs and make sure they get preferred over their older versions on import, for example by setting up PYTHONPATH.

Alternatively it should also be possible to install the package using easy_install, which would automatically install its dependencies including ZODB 3.8, too. Again you would need to set up your PYTHONPATH to make sure the desired versions are used. However, installing the package like this is likely to have side effects on other Zope/Plone instances on your system, so you probably want to use virtualenv here at least.

Overall, to get started without too much pain, a buildout-based installation is recommended — for example the provided buildout.

Will this be available for Plone 2.5.x?

Yes, support for the 2.5 series is planned and next on the agenda.

What about image support, i.e. a drop-in for ATImage content?

While just replacing the primary field in ATImage’s schemata should probably already work quite well, proper image support is planned for a later release. “proper” here means using a sub-typing approach as presented by Rocky Burt in Naples, which will have several advantages including a cleaner and better structured code, but will also take a little longer to implement.

Strange messages like Exception exceptions.OSError: (2, 'No such file or directory', '.../tmpZvxjZB') in <bound method _TemporaryFileWrapper.__del__ of <closed file '<fdopen>', mode 'w+b' at 0x7317650>> ignored get written to the logs whenever a file is uploaded. Is that an error or something to worry about?

No, that’s fine, it’s just a small annoyance, that should be fixed eventually. In case you care, the problem is that the zope publisher creates a temporary file for each upload it receives. Once the upload has finished that temporary file is passed to the blob machinery, which moves it into its blob storage. However, at the end of the request the wrapper class for temporary files tries to remove the file as well, since well, it’s supposed to be temporary. At that time the file is already gone though, and the above warning is issued.

Feedback

Any kind of feedback like bug reports, suggestions, feature requests and most preferably success stories is most welcome and much appreciated. Especially, it would be interesting to hear about success or problems with migration of existing content and installations on platforms other than OSX.

So please feel free to post comments to the above mentioned plone enhancement ticket, file tickets in the issue tracker or contact me on #plone, through the plone developer mailing list or directly via email.