collective.indexing 1.0b4

Introduction

collective.indexing is an approach to provide an abstract framework for queuing and optimizing index operations in Plone as well as dispatching them to various backends. The default implementation aims to replace the standard indexing mechanism of CMF to allow index operations to be handled asynchronously in a backwards-compatible way.

Queuing these operations on a transaction level allows to get rid of redundant indexing of objects and thereby providing a substantial performance improvement. By leveraging the component architecture and event system of zope3 collective.indexing also makes it much easier to use backends other than or in addition to the standard portal catalog for indexing, such as dedicated search engine solutions like Solr, Xapian or Google Search Appliance. One backend implementation designed to be used with this package has already been started in the form of collective.solr.

If you are using CacheSetup make sure that you have CMFSquidTool 1.5 or later: older versions do not handle purge requests being generated during transaction commit and will produce internal errors.

Current Status

The implementation is considered to be nearly finished. The package can be installed in a Plone 3.x site to enable indexing operations to be queued, optimized and dispatched to the standard portal catalog on the zope transaction boundary thereby improving the Plone’s out-of-the-box performance. A buildout is provided for your convenience.

At the moment the package requires “monkey patches” to the mixin classes currently used to hook up indexing, i.e. CMFCatalogAware (from CMF) and CatalogMultiplex (from Archetypes). It is planned to make these classes “pluggable” by use of adapters, allowing collective.indexing to hook in in clean ways. This will be proposed as a PLIP for inclusion into Plone 3.2.

In conjunction with collective.solr the package also provides a working solution for integration of Solr with Plone. Based on a schema configurable at zc.buildout level indexing operations can be dispatched a Solr instance in addition or alternatively to the standard catalog. This allows for minimal and very efficient indexing of standard Plone content items based on Archetypes. Providing support for other content types is rather trivial and will be support soon.

The code was written with emphasis on minimalism, clarity and maintainability. It comes with extensive tests covering the code base at more than 95%.

For outstanding issues and features remaining to be implemented please see the to-do list included in the package.

Credits

This code was inspired by enfold.indexing and enfold.solr by Enfold Systems as well as work done at the snowsprint’08. The TransactionManager pattern is taken from enfold.solr. Development was kindly sponsored by Elkjop.

1.0b4 - Released June 30th, 2008

• Perform processing of the queue during “active” state of the transaction as additional changes are forbidden in “committing” state. Those changes can for example be caused by indexes writing back data to content items, such as the modification time. [witsch, mj]

1.0b3 - Released June 18th, 2008

• Fix an issue where objects providing their own __setattr__ could potentially not be deleted. [witsch]

1.0b2 - Released June 2nd, 2008

• Add aq_inner to prevent infinite recursion with safe_hasattr. [witsch]

1.0b1 - Released May 28, 2008

• Make sure we get REQUEST correctly in PathWrapper. [tesdal]

1.0a3 - Released May 28, 2008

• Respect overridden indexing methods to prevent erroneous indexing and generally allow special handling. [witsch]

• Fix leftover index entry after renaming an object. [witsch]

• The bad monkey smacked back with a vengeance, but was finally tamed. [witsch]

• Smacked a bad monkey that was checking for nonexisting attribute getObjPositionInParent. [tesdal]

1.0a2 - Released May 25, 2008

• Fix bug regarding different types in the queue reducer logic. [fschulze]

• Fixed renaming of content items by replacing PloneTool.reindexOnReorder with a saner version that doesn’t rely on the catalog. [witsch]

• Various fixes, cleanups and optimizations. [witsch]

• Fixed monkey patches so that normal indexing remains functional when queued indexing has been deactivated (or the GS profile had not been applied yet). [witsch]

1.0a1 - Released March 31, 2008

• Initial release [tesdal, witsch]