Archetypes storage for storing fields raw values on the file system.
Project description
By Ingeniweb.
1 About FileSystemStorage
FileSystemStorage (FSS) is an Archetypes storage for storing fields raw values on the file system. This storage is used to avoid unnecessary growth of the ZODB’s FileStorage (Data.fs) when using a lot of large files.
Please note that FSS is a Plone component for content types developers. Do not expect anything more in your Plone site if you don’t use Plone components that may use FSS, such as AttachmentField.
2 Copyright and license
Copyright (c) 2005 - 2008 Ingeniweb SAS
This software is subject to the provisions of the GNU General Public License, Version 2.0 (GPL). A copy of the GPL should accompany this distribution. THIS SOFTWARE IS PROVIDED “AS IS” AND ANY AND ALL EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE
See the LICENSE file that comes with this product.
3 Requirements
4 Upgrading
4.1 From FSS 2.5.x and older
The way we configure the storage and backup paths as well as the storage strategies HAVE CHANGED IN DEPTH. There’s no automatic migration from this version. Please create a configuration file that clones your old configlet preferences as indicated in FSS main configuration.
4.2 From iw.fss 2.7.1
From this version and later version, iw.fss registers upgrade steps in GenericSetup. When upgrading to a newer version of iw.fss, just have a look at the “Upgrade” tab of the portal_setup tool in your Plone site.
Warning, due to the removal of the portal_fss tool upgrading from 2.7.1 does not retain the RDF options (activated and specific script). If you used RDF with 2.7.1, you need to redo the same RDF setting in the FileSystemStorage config panel.
5 Migrating between strategies
Use the bin/strategymigrator.py shell utility that ships with FSS. Get more info on this utility using the –help option.
Note that only flat -> directory and directory -> flat are available today. Sponsorship is welcome - as indicated in Support and feedback - if you need other strategy migrations.
6 Installation
6.1 With buildout
This example speaks of itself:
[buildout] ... eggs = ... iw.fss ... [instance] ... zcml = ... iw.fss iw.fss-meta ...
You may also use the iw.recipe.fss recipe egg to configure fully an instance.
6.2 From SVN reposo
Inflate iw.fss package into your zope instance lib/python directory. I’m sure it’s already done.
Add $INSTANCE_HOME/etc/package-includes/iw.fss-meta.zcml file with this line:
<include package="iw.fss" file="meta.zcml"/>
Add $INSTANCE_HOME/etc/package-includes/iw.fss-configure.zcml file with this line:
<include package="iw.fss" />
6.3 Configuring your Plone sites
Read carefully Storage strategies and FSS main configuration below and configure your instance in plone-filesystemstorage.conf accordingly. You may use the iw.recipe.fss recipe in your buildout.cfg to do this.
Start your Zope instance
In ZMI, go into your plone site and install the product using portal_quickinstaller
Optionally, tweak some - self explained - preferences in the FileSystemStorage configuration panel of your Plone site.
7 Storage strategies
A storage strategy defines how field values will be stored in your files system.
7.1 Strategies overview
FSS comes with 4 storage strategies. Each strategy requires two directories:
The storage directory stores the values of relevant fields in files according to the selected strategy.
The backup directory that keeps the values of the fields of deleted contents for “Undo” purpose.
Only one storage strategy can be selected on a given Plone site, but different Plone site in the same Zope instance may have distinct storage strategies.
You need the best performances: choose the Flat storage strategy or Directory storage strategy.
You need a storage that looks like your Plone site (in example to publish the storage directory in a read only samba or NFS share): choose the Site storage strategy 1 or the Site storage strategy 2.
If you care about CMFEdition support (means that one of the content types using FSS is versioned with CMFEditions), you should not use the Site storage strategy 1 or the Site storage strategy 2.
See FSS main configuration to configure the best suited strategy on your Zope instance and Plone sites.
7.2 Flat storage strategy
All field values are stored in a flat structure. This strategy is the default one.
Filename of these stored values: <uid of content>_<field name>
Filename of rdf file: <uid of content>_<field name>.rdf
Filename of backup values: <uid of content>_<field name>.bak
Rdf files are not backed up. They are automatically generated.
Example of storage:
fssDirectory | |- f42ad00adb7d4580f929d734bf1ed3ab_image | |- f42ad00adb7d4580f929d734bf1ed3ab_image.rdf | |- f42ad00adb7d4580f929d734bf1ed3ab_file | |- f42ad00adb7d4580f929d734bf1ed3ab_file.rdf fssBackupDirectory | |- 9efeb7638fb35f5f9b9955da3e2dbfec_file.bak
7.3 Directory storage strategy
All field values are stored in a directory structure. Sub directories are defined on two level. First level of directory uses the 2 characters of content uid. Second level of directory uses the 4 characters of content uid. Backup files are stored in a flat structure.
Filename of these stored values: <uid of content>_<field name>
Filename of rdf file: <uid of content>_<field name>.rdf
Filename of backup values: <uid of content>_<field name>.bak
RDF files are not backed up. They are automatically generated
Example of storage:
fssDirectory
|
|- f42
|
|- f42ad
|
|- f42ad00adb7d4580f929d734bf1ed3ab_image
|
|- f42ad00adb7d4580f929d734bf1ed3ab_image.rdf
|
|- f42ad00adb7d4580f929d734bf1ed3ab_file
|
|- f42ad00adb7d4580f929d734bf1ed3ab_file.rdf
fssBackupDirectory
|
|- 9efeb7638fb35f5f9b9955da3e2dbfec_file.bak
Depending in one hand on your file system performances facing a huge amount of files in the same directory, and in the other hand on the number of contents relying on FSS, you might choose this strategy or the Flat storage strategy.
7.4 Site storage strategy 1
All field values are stored in a directory structure mirroring structure of PloneSite. Backup files are stored in a flat structure.
Filename of these stored values: Filename of field value or field name if not defined
Filename of rdf file: <field name>.rdf
Filename of backup values: <uid of content>_<field name>.bak
Rdf files are not backed up. They are automatically generated
Example of storage:
fssDirectory
|
|- members
|
|- john
|
|- dummy-document
|
|- image
| |
| |- moutain.jpg
| |
| |- image.rdf
|
|- file
|
|- diary.odt
|
|- file.rdf
fssBackupDirectory
|
|- 9efeb7638fb35f5f9b9955da3e2dbfec_file.bak
7.5 Site storage strategy 2
All field values are stored in a directory structure mirroring structure of PloneSite. Backup files are stored in a flat structure.
Filename of these stored values: Filename of field value or field name if not defined
Filename of rdf file: <field filename>.rdf
Filename of backup values: <uid of content>_<field name>.bak
Rdf files are not backed up. They are automatically generated
Example of storage:
fssDirectory
|
|- members
|
|- john
|
|- dummy-document
|
|- fss.cfg
|
|- moutain.jpg
|
|- mountain.jpg.rdf
|
|- diary.odt
|
|- diary.odt.rdf
fssBackupDirectory
|
|- 9efeb7638fb35f5f9b9955da3e2dbfec_file.bak
8 FSS main configuration
FSS is mainly configured the ZConfig way. At startup, the configuration file will be searched (in that order) in:
$INSTANCE_HOME/etc/plone-filesystemstorage.conf
/path/to/iw/fss/etc/plone-filesystemstorage.conf
/path/to/iw/fss/etc/plone-filesystemstorage.conf.in
A sample working configuration in provided in this last file. It assumes you have $INSTANCE_HOME/var/fss_storage and $INSTANCE_HOME/var/fss_backup directories, both being read/write enabled to the user that runs the Zope process, unless Zope will raise an error at startup.
All configuration doc you need is in plone-filesystemstorage.conf.in comments.
Inconsistent configuration features raise explicit error messages at Zope startup.
Note that we didn’t include this in zope.conf in order to keep Zope 2.7 and Zope 2.8 compatibility.
9 Configuration panel
9.1 FSS and RDF
As mentioned above in Storage strategies, FSS can optionally store RDF files in the back-end storage. These RDF files conform the DCMES-XML standard on XML expression of DublinCore elements.
If you select the Flat storage strategy, the RDF files may be used to build a files tree as close as possible of the Plone tree structure from the storage back-end, with the build_fs_tree.py utility:
$ cd /path/to/iw/fss/bin $ python build_fs_tree.py --help
9.2 FSS maintenance
The “Maintenance” tab of FSS configuration panel shows some statistics about the amount of files managed by FSS.
In addition 2 additional buttons are provided:
Update FSS files : Cleans up the back-end storage directories. Files not referenced from a content object are removed.
Update RDF files : Builds all RDF files, in the event you chose to select the generetion of RDF files after having inserted relevant contents.
10 Zope backups
Of course, you need to synch your Data.fs backups with the storage paths backups, unless you may restore corrupted/incomplete sites.
11 Caveats and pittfalls
Note that whatever’s the selected strategies, DO NEVER CHANGE ANY FILE IN THE STORAGE PATHS UNLESS YOU KNOW WHAT YOU DO, otherwise you’ll loose your contents, your job, your money, your friends, your wife/husband and children.
DO NEVER CHANGE THE STRATEGY FOR A PLONE SITE ONCE THIS SITE HAS FIELDS STORED IN ITS FSS STORAGE PATH unless you ran the appropriate strategy migrator - see Migrating between strategies otherwise you’re loose your contents, your job, …
Thought it is possible to share the same storage path and backup path within various Plone sites, since you don’t mind of your data in development sites, you should really configure FSS such each Plone site has its own private storage path and backup path. In cas you ignore this warning, DO NEVER CLICK ANY BUTTON IN THE MAINTENANCE TAB OF THE FSS CONFIGURATION PANEL otherwise you’ll loose your contents, your job, …
Zexp exports don’t embed the associated FSS storage. So don’t move your Plone site within Zope instances using zexp exports unless you move the storage and backup directories along with the zexp file.
After changing your configuration (see FSS main configuration), always restart your Zope instance in foreground since configuration errors are not reported when Zope is started as daemon or Windows service.
12 Developer’s corner
12.1 Using FSS in your content types
We assume that creating AT based content types is a familiar activity.
When creating the schema of a content type, you usually don’t mention the storage of your fields since AT provides a default ZODB storage. You can choose another storage, either another one that ships in the Archetypes bundle, or a third party storage like FSS.
Here is a small example that doesn’t need much comments:
# Usual Zope/CMF/Plone/Archetypes imports
...
from iw.fss.FileSystemStorage import FileSystemStorage
...
my_schema = Schema((
FileField('file',
...
storage=FileSystemStorage(),
widget=FileWidget(...)
),
...
))
...
You may have a look at examples/FSSItem.py to see a demo content type using FSS. If you want to play with this content type:
$ $INSTANCE_HOME/bin/zopectl stop $ export FSS_INSTALL_EXAMPLE_TYPES=1 $ $INSTANCE_HOME/bin/zopectl start
12.2 Monkey patching a third party content type
12.2.1 The Python way
This works with any Zope version listed in the Requirements. If you run a Zope 2.9 or later instance, you should prefer The ZCML way.
This simple example shows how to plug the ATFile standard type from any custom product. Let’s say we’re in the __init__.py of your custom product:
... from iw.fss import zcml as fss_utils from Products.ATContentTypes import atct ... fss_utils.patchATType(atct.ATFile, ['file']) ...
12.2.2 The ZCML way
This works only with Zope 2.9 and up, and is the recommanded way to wire your AT based content types with FSS since this way hides the API details that may be deprecated in the future.
You just need to add the “fss” namespace support and some directives in your “configure.zcml” like this to have the same effect as in the example of The Python way above:
<configure
...
xmlns:zcml="http://namespaces.zope.org/zcml"
xmlns:fss="http://namespaces.ingeniweb.com/filesystemstorage"
...>
...
<fss:typeWithFSS
zcml:condition="installed iw.fss"
class="Products.ATContentTypes.atct.ATFile"
fields="file" />
...
</configure>
The (required) attributes of this directive are:
class: the dotted name of the AT based content type class.
fields: one or more (space separated) field names to wire with FSS.
12.3 Storing Plone standard content types with FSS
Include this ZCML element either in your instance package-includes directory or in any personal configure.zcml in your instance packages:
<include package="iw.fss" file="atct.zcml" />
This ZCML file wires FSS with the relevant content types that ship with Plone bundle:
File
Image
News Item
12.4 Customize RDF info set
You can add data to the RDF info set via a FSS dedicated - and optional - hook.
Open the FSS configuration panel and give the name of your personal RDF data script, let’s call it fss_custom_rdf for this example. This script will add or change the RDF default data set.
12.4.1 RDF script interface
The script may be in any Plone layer like custom or any other in the skins path.
Such script is expected to have 4 parameters:
name: the name of the field
instance: the content object being processed
properties: FSS field properties mapping, depending on the type of field associated to FSS storage.
default_rdf: a mapping structure of the RDF data, including the namespaces, and the elements.
Such script is expected to return an updated default_rdf mapping structure.
Please read the source of classes FSSInfo, FSSFileInfo and FSSImageInfo from module FileSystemStorage.py, RDFWriter from module rdf.py for detailed information on formats of expected data.
12.4.2 Simple example
We need to add the date of RDF file generation in an element like this:
<fss:rdf_date>2007-05-13 13:21:00</fss:rdf_date>
Here’s the script:
## Script (Python) "fss_custom_rdf"
##bind container=container
##bind context=context
##bind namespace=
##bind script=script
##bind subpath=traverse_subpath
##parameters=name, instance, properties, default_rdf
##title=
##
"""We add the date of RDF file generation"""
rdf_args = default_rdf.copy()
# Add the date of RDF generation
new_props = []
new_props.extend(rdf_args['field_props'])
new_props.append({'id': 'fss:rdf_date', 'value': DateTime().ISO()})
rdf_args['field_props'] = new_props
return rdf_args
12.4.3 More complex example
In this more complex (and stupid - don’t do this ;) example, we remove the elements from fss namespace and simulate the standard behaviour for other elements (dc and rdf elements).
Here’s the code:
## Script (Python) "fss_custom_rdf"
##bind container=container
##bind context=context
##bind namespace=
##bind script=script
##bind subpath=traverse_subpath
##parameters=name, instance, properties, default_rdf
##title=
##
from Products.CMFCore.utils import getToolByName
rdf_args = {}
# Set RDF charset
ptool = getToolByName(instance, 'portal_properties')
rdf_args['charset'] = ptool.site_properties.default_charset
# Set RDF namespaces
rdf_args['namespaces'] = (
{'id': 'xmlns:rdf', 'value': 'http://www.w3.org/1999/02/22-rdf-syntax-ns#'},
{'id': 'xmlns:dc', 'value': 'http://purl.org/dc/elements/1.1/'},
{'id': 'xmlns:fss', 'value': 'http://namespace.ingeniweb.com/fss'},)
# Set field url
utool = getToolByName(instance, 'portal_url')
portal_path = utool.getPortalObject().getPhysicalPath()
portal_path_len = len(portal_path)
rel_path = '/'.join(instance.getPhysicalPath()[portal_path_len:])
fss_path = '%s/%s' % (rel_path, name)
rdf_args['field_url'] = fss_path
# Set RDF properties
props = (
{'id': 'dc:title', 'value': instance.title_or_id()},
{'id': 'dc:description', 'value': instance.Description()},
{'id': 'dc:language', 'value': instance.Language()},
{'id': 'dc:creator', 'value': instance.Creator()},
{'id': 'dc:date', 'value': instance.modified()},
{'id': 'dc:format', 'value': properties.get('mimetype', 'text/plain')},
)
rdf_args['field_props'] = props
# Returns the RDF structure
return rdf_args
12.4.4 Recommanded practices on custom RDF
Do not change the existing elements under fss namespace unless the bin/build_fs_tree.py utility won’t work. In other terms, do not use the script of the More complex example.
Adding or changing elements to the dc and rdf namespaces may screw up external utilities based on canonical DCMES-XML standard.
Consider adding your own namespace for your custom extra elements.
13 Testing
Please read ./tests/README.txt.
14 Other documentation
See ./doc/* and ./locales/README.txt
15 Downloads
You may find newer stable versions of FSS and pointers to related informations (tracker, doc, …) from http://pypi.python.org/pypi/iw.fss
16 Subversion repository
Stay in tune with the freshest (maybe unstable) versions or participate to the FileSystemStorage evolutions:
17 Support and feedback
Please read all the documentation that comes with this product before asking for support, unless you might get a RTFM reply ;)
Localisation issues - other than french - should be reported to the relevant translators (see Credits below).
Report bugs using the tracker (the Tracker link from http://plone.org/iw.fss). Please provide in your bug report:
Your configuration (Operating system+Zope+Plone+Products/versions).
The storage strategy in use.
The full traceback if available.
One or more scenario that triggers the bug.
Note that we do not support bug reports on Subversion trunk or branches checkouts.
Mail to Ingeniweb support in English or French to ask for specific support.
18 Credits
Main developer: Cyrille Lebeaupin
ZConfig and ZCML support: Gilles Lenfant
HTTP “Range” support: Youenn Boussard
Paris 2008 sprint additional contributors: - Mehdi Benammar - Romain Griffiths
See locales/README.txt about localization credits.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
