RelStorage 1.4.0

Configuring Your Database

You need to configure a database and user account for RelStorage. RelStorage will populate the database with its schema the first time it connects.

$ sudo su - postgres
$ createuser --pwprompt zodbuser
$ createdb -O zodbuser zodb

local  zodb  zodbuser  md5

/etc/init.d/postgresql reload

$ mysql -u root -p

CREATE USER 'zodbuser'@'localhost' IDENTIFIED BY 'mypassword';
CREATE DATABASE zodb;
GRANT ALL ON zodb.* TO 'zodbuser'@'localhost';
FLUSH PRIVILEGES;

$ su - oracle
$ sqlplus / as sysdba

CREATE USER zodb IDENTIFIED BY mypassword;
GRANT CONNECT, RESOURCE, CREATE TABLE, CREATE SEQUENCE TO zodb;
GRANT EXECUTE ON DBMS_LOCK TO zodb;

Configuring Plone

To install RelStorage in Plone, see the instructions in the following article:

http://shane.willowrise.com/archives/how-to-install-plone-with-relstorage-and-mysql/

Plone uses the plone.recipe.zope2instance Buildout recipe to generate zope.conf, so the easiest way to configure RelStorage in a Plone site is to set the rel-storage parameter in buildout.cfg. The rel-storage parameter contains settings separated by newlines, with these values:

• type: any database type supported (postgresql, mysql, or oracle)

• RelStorage options like cache-servers and poll-interval

• Adapter-specific options

An example:

rel-storage =
    type mysql
    db plone
    user plone
    passwd PASSWORD

Configuring Zope 2

To integrate RelStorage in Zope 2, specify a RelStorage backend in etc/zope.conf. Remove the main mount point and add one of the following blocks. For PostgreSQL:

%import relstorage
<zodb_db main>
  mount-point /
  <relstorage>
    <postgresql>
      # The dsn is optional, as are each of the parameters in the dsn.
      dsn dbname='zodb' user='username' host='localhost' password='pass'
    </postgresql>
  </relstorage>
</zodb_db>

For MySQL:

%import relstorage
<zodb_db main>
  mount-point /
  <relstorage>
    <mysql>
      # Most of the options provided by MySQLdb are available.
      # See component.xml.
      db zodb
    </mysql>
  </relstorage>
</zodb_db>

For Oracle (10g XE in this example):

%import relstorage
<zodb_db main>
  mount-point /
  <relstorage>
    <oracle>
      user username
      password pass
      dsn XE
    </oracle>
 </relstorage>
</zodb_db>

To add ZODB blob support, provide a blob-dir parameter that specifies where to store the blobs. For example:

%import relstorage
<zodb_db main>
  mount-point /
  blob-dir ./blobs
  <relstorage>
    <postgresql>
      dsn dbname='zodb' user='username' host='localhost' password='pass'
    </postgresql>
  </relstorage>
</zodb_db>

Configuring repoze.zodbconn

To use RelStorage with repoze.zodbconn, a package that makes ZODB available to WSGI applications, create a configuration file with contents similar to the following:

%import relstorage
<zodb main>
  <relstorage>
    <mysql>
      db zodb
    </mysql>
  </relstorage>
  cache-size 100000
</zodb>

repoze.zodbconn expects a ZODB URI. Use a URI of the form zconfig://path/to/configuration#main.

The zodbconvert Utility

RelStorage comes with a script named zodbconvert that converts databases to different formats. Use it to convert a FileStorage instance to RelStorage and back, or to convert between different kinds of RelStorage instances, or to convert other kinds of storages that support the storage iterator protocol.

When converting between two history-preserving databases (note that FileStorage uses a history-preserving format), zodbconvert utility preserves all objects and transactions, meaning you can still use the ZODB undo feature after the conversion, and you can convert back using the same process in reverse. When converting from a history-free database to either a history-free database or a history-preserving database, zodbconvert retains all data, but the converted transactions will not be undoable. When converting from a history-preserving storage to a history-free storage, zodbconvert drops all historical information during the conversion.

<filestorage source>
  path /zope/var/Data.fs
</filestorage>

<relstorage destination>
  <mysql>
    db zodb
  </mysql>
</relstorage>

Migrating to a new version of RelStorage

Sometimes RelStorage needs a schema modification along with a software upgrade. Hopefully, this will not often be necessary.

No schema migration is required if you are using PostgreSQL or MySQL and upgrading from version 1.1.2 or later. See the notes subdirectory if you are upgrading from an older version.

To migrate Oracle to version 1.4, see migrate-to-1.4.txt.

PostgreSQL Adapter Options

The PostgreSQL adapter accepts:

dsn

Specifies the data source name for connecting to PostgreSQL. A PostgreSQL DSN is a list of parameters separated with whitespace. A typical DSN looks like:

dbname='zodb' user='username' host='localhost' password='pass'

If dsn is omitted, the adapter will connect to a local database with no password. Both the user and database name will match the name of the owner of the current process.

MySQL Adapter Options

The MySQL adapter accepts most parameters supported by the MySQL-python library, including:

host

string, host to connect

user

string, user to connect as

passwd

string, password to use

db

string, database to use

port

integer, TCP/IP port to connect to

unix_socket

string, location of unix_socket (UNIX-ish only)

conv

mapping, maps MySQL FIELD_TYPE.* to Python functions which convert a string to the appropriate Python type

connect_timeout

number of seconds to wait before the connection attempt fails.

compress

if set, gzip compression is enabled

named_pipe

if set, connect to server via named pipe (Windows only)

init_command

command which is run once the connection is created

read_default_file

see the MySQL documentation for mysql_options()

read_default_group

see the MySQL documentation for mysql_options()

client_flag

client flags from MySQLdb.constants.CLIENT

load_infile

int, non-zero enables LOAD LOCAL INFILE, zero disables

Oracle Adapter Options

The Oracle adapter accepts:

user

The Oracle account name

password

The Oracle account password

dsn

The Oracle data source name. The Oracle client library will normally expect to find the DSN in /etc/oratab.

Options for zodbpack

--days or -d

Specifies how many days of historical data to keep. Defaults to 0, meaning no history is kept. This is meaningful even for history-free storages, since unreferenced objects are not removed from the database until the specified number of days have passed.

1.4.0 (2010-09-30)

• Made compatible with ZODB 3.10.0b7.

• Enabled ketama and compression in pylibmc_wrapper. Both options are better for clusters. [Helge Tesdal]

• Oracle: Use a more optimal query for POSKeyError logging. [Helge Tesdal]

• Fixed a NameError that occurred when getting the history of an object where transaction extended info was set. [Helge Tesdal]

1.4.0c4 (2010-09-17)

• Worked around an Oracle RAC bug: apparently, in a RAC environment, the read-only transaction mode does not isolate transactions in the manner specified by the documentation, so Oracle users now have to use serializable isolation like everyone else. It’s slower but more reliable.

• Use the client time instead of the database server time as a factor in the transaction ID. RelStorage was using the database server time to reduce the need for synchronized clocks, but in practice, that policy broke tests and did not really avoid the need to synchronize clocks. Also, the effect of unsynchronized clocks is predictable and manageable: you’ll get bunches of transactions with sequential timestamps.

• If the database returns an object from the future (which should never happen), generate a ReadConflictError, hopefully giving the application a chance to recover. The most likely causes of this are a broken database and threading bugs.

1.4.0c3 (2010-07-31)

• Always update the RelStorage cache when opening a database connection for loading, even when no ZODB Connection is using the storage. Otherwise, code that used the storage interface directly could cause the cache to fall out of sync; the effects would be seen in the next ZODB.Connection.

• Added a ZODB monkey patch that passes the “force” parameter to the sync method. This should help the poll-interval option do its job better.

1.4.0c2 (2010-07-28)

• Fixed a subtle bug in the cache code that could lead to an AssertionError indicating a cache inconsistency. The inconsistency was caused by after_poll(), which was ignoring the randomness of the order of the list of recent changes, leading it to sometimes put the wrong transfer ID in the “delta_after” dicts. Also expanded the AssertionError with debugging info, since cache inconsistency can still be caused by database misconfiguration and mismatched client versions.

• Oracle: updated the migration notes. The relstorage_util package is not needed after all.

1.4.0c1 (2010-06-19)

• History-preserving storages now replace objects on restore instead of just inserting them. This should solve problems people were having with the zodbconvert utility.

• Oracle: call the DBMS_LOCK.REQUEST function directly instead of using a small package named relstorage_util. The relstorage_util package was designed as a secure way to access the DBMS_LOCK package, but the package turned out to be confusing to DBAs and provided no real security advantage. People who have already deployed RelStorage 1.4.x on Oracle need to do the following:

  GRANT EXECUTE ON DBMS_LOCK TO <zodb_user>;

  You can also drop the relstorage_util package. Keep the relstorage_op package.

• Made compatible with ZODB 3.10.

• MySQL: specify the transaction isolation mode for every connection, since the default is apparently not necessarily “read committed” anymore.

1.4.0b3 (2010-02-02)

• Auto-reconnect in new_oid().

1.4.0b2 (2010-01-30)

• Include all test subpackages in setup.py.

• Raise an error if MySQL reverts to MyISAM rather than using the InnoDB storage engine.

1.4.0b1 (2009-11-17)

• Added the keep-history option. Set it to false to keep no history. (Packing is still required for garbage collection and blob deletion.)

• Added the replica-conf and replica-timeout options. Set replica-conf to a filename containing the location of database replicas. Changes to the file take effect at transaction boundaries.

• Expanded the option documentation in README.txt.

• Revised the way RelStorage uses memcached. Minimized the number of trips to both the cache server and the database.

• Added an in-process pickle cache that serves a function similar to the ZEO cache.

• Added a wrapper module for pylibmc.

• Store operations now use multi-insert and multi-delete SQL statements to reduce the effect of network latency.

• Renamed relstorage.py to storage.py to overcome import issues. Also moved the Options class to options.py.

• Updated the patch for ZODB 3.7 and 3.8 to fix an issue with blobs and subtransactions.

• Divided the implementation of database adapters into many small objects, making the adapter code more modular. Added interfaces that describe the duties of each part.

• Oracle: Sped up restore operations by sending short blobs inline.

• Oracle: Use a timeout on commit locks. This requires installation of a small PL/SQL package that can access DBMS_LOCK. See README.txt.

• Oracle: Used PL/SQL bulk insert operations to improve write performance.

• PostgreSQL: use the documented ALTER SEQUENCE RESTART WITH statement instead of ALTER SEQUENCE START WITH.

• Moved MD5 sum computation to the adapters so they can choose not to use MD5.

• Changed loadSerial to load from the store connection only if the load connection can not provide the object requested.

• Stopped wrapping database disconnect exceptions. Now the code catches and handles them directly.

• Use the store connection rather than the load connection for OID allocation.

• Detect and handle backward time travel, which can happen after failover to an out-of-date asynchronous slave database. For simplicity, invalidate the whole ZODB cache when this happens.

• Replaced the speed test script with a separately distributed package, zodbshootout.

• Added the zodbpack script.

1.3.0b1 (2009-09-04)

• Added support for a blob directory. No BlobStorage wrapper is needed. Cluster nodes will need to use a shared filesystem such as NFS or SMB/CIFS.

• Added the blob-dir parameter to the ZConfig schema and README.txt.

1.2.0 (2009-09-04)

• In Oracle, trim transaction descriptions longer than 2000 bytes.

• When opening the database for the first time, don’t issue a warning about the inevitable POSKeyError on the root OID.

• If RelStorage tries to unpickle a corrupt object state during packing, it will now report the oid and tid in the log.

1.2.0b2 (2009-05-05)

• RelStorage now implements IMVCCStorage, making it compatible with ZODB 3.9.0b1 and above.

• Removed two-phase commit support from the PostgreSQL adapter. The feature turned out to be unnecessary.

• Added MySQL 5.1.34 and above to the list of supportable databases.

• Fixed minor test failures under Windows. Windows is now a supportable platform.