ZEO 5.2.1.dev0

Client cache

Each ZEO client keeps an on-disk cache of recently used data records to avoid fetching those records from the server each time they are requested. It is usually faster to read the objects from disk than it is to fetch them over the network. The cache can also provide read-only copies of objects during server outages.

The cache may be persistent or transient. If the cache is persistent, then the cache files are retained for use after process restarts. A non-persistent cache uses temporary files that are removed when the client storage is closed.

The client cache size is configured when the ClientStorage is created. The default size is 20MB, but the right size depends entirely on the particular database. Setting the cache size too small can hurt performance, but in most cases making it too big just wastes disk space.

ZEO uses invalidations for cache consistency. Every time an object is modified, the server sends a message to each client informing it of the change. The client will discard the object from its cache when it receives an invalidation. (It’s actually a little more complicated, but we won’t get into that here.)

Each time a client connects to a server, it must verify that its cache contents are still valid. (It did not receive any invalidation messages while it was disconnected.) This involves asking the server to replay invalidations it missed. If it’s been disconnected too long, it discards its cache.

Invalidation queue

The ZEO server keeps a queue of recent invalidation messages in memory. When a client connects to the server, it sends the timestamp of the most recent invalidation message it has received. If that message is still in the invalidation queue, then the server sends the client all the missing invalidations.

The default size of the invalidation queue is 100. If the invalidation queue is larger, it will be more likely that a client that reconnects will be able to verify its cache using the queue. On the other hand, a large queue uses more memory on the server to store the message. Invalidation messages tend to be small, perhaps a few hundred bytes each on average; it depends on the number of objects modified by a transaction.

You can also provide an invalidation age when configuring the server. In this case, if the invalidation queue is too small, but a client has been disconnected for a time interval that is less than the invalidation age, then invalidations are replayed by iterating over the lower-level storage on the server. If the age is too high, and clients are disconnected for a long time, then this can put a lot of load on the server.

Transaction timeouts

A ZEO server can be configured to timeout a transaction if it takes too long to complete. Only a single transaction can commit at a time; so if one transaction takes too long, all other clients will be delayed waiting for it. In the extreme, a client can hang during the commit process. If the client hangs, the server will be unable to commit other transactions until it restarts. A well-behaved client will not hang, but the server can be configured with a transaction timeout to guard against bugs that cause a client to hang.

If any transaction exceeds the timeout threshold, the client’s connection to the server will be closed and the transaction aborted. Once the transaction is aborted, the server can start processing other client’s requests. Most transactions should take very little time to commit. The timer begins for a transaction after all the data has been sent to the server. At this point, the cost of commit should be dominated by the cost of writing data to disk; it should be unusual for a commit to take longer than 1 second. A transaction timeout of 30 seconds should tolerate heavy load and slow communications between client and server, while guarding against hung servers.

When a transaction times out, the client can be left in an awkward position. If the timeout occurs during the second phase of the two phase commit, the client will log a panic message. This should only cause problems if the client transaction involved multiple storages. If it did, it is possible that some storages committed the client changes and others did not.

Connection management

A ZEO client manages its connection to the ZEO server. If it loses the connection, it attempts to reconnect. While it is disconnected, it can satisfy some reads by using its cache.

The client can be configured with multiple server addresses. In this case, it assumes that each server has identical content and will use any server that is available. It is possible to configure the client to accept a read-only connection to one of these servers if no read-write connection is available. If it has a read-only connection, it will continue to poll for a read-write connection.

If a single address resolves to multiple IPv4 or IPv6 addresses, the client will connect to an arbitrary of these addresses.

SSL

ZEO supports the use of SSL connections between servers and clients, including certificate authentication. We’re still understanding use cases for this, so details of operation may change.

Server quick-start/ad-hoc operation

You can quickly start a ZEO server from a Python prompt:

import ZEO
address, stop = ZEO.server()

This runs a ZEO server on a dynamic address and using an in-memory storage.

We can then create a ZEO client connection using the address returned:

connection = ZEO.connection(addr)

This is a ZODB connection for a database opened on a client storage instance created on the fly. This is a shorthand for:

db = ZEO.DB(addr)
connection = db.open()

Which is a short-hand for:

client_storage = ZEO.client(addr)

import ZODB
db = ZODB.db(client_storage)
connection = db.open()

If you exit the Python process, the storage exits as well, as it’s run in an in-process thread.

You shut down the server more cleanly by calling the stop function returned by the ZEO.server function.

To have data stored persistently, you can specify a file-storage path name using a path parameter. If you want blob support, you can specify a blob-file directory using the blob_dir directory.

You can also supply a port to listen on, full storage configuration and ZEO server configuration options to the ZEO.server function. See it’s documentation string for more information.

Server configuration

The script runzeo runs the ZEO server. The server can be configured using command-line arguments or a configuration file. This document only describes the configuration file. Run runzeo -h to see the list of command-line arguments.

The configuration file specifies the underlying storage the server uses, the address it binds to, and a few other optional parameters. An example is:

<zeo>
  address zeo.example.com:8090
</zeo>

<filestorage>
  path /var/tmp/Data.fs
</filestorage>

<eventlog>
  <logfile>
    path /var/tmp/zeo.log
    format %(asctime)s %(message)s
  </logfile>
</eventlog>

The format is similar to the Apache configuration format. Individual settings have a name, 1 or more spaces and a value, as in:

address zeo.example.com:8090

Settings are grouped into hierarchical sections.

The example above configures a server to use a file storage from /var/tmp/Data.fs. The server listens on port 8090 of zeo.example.com. The ZEO server writes its log file to /var/tmp/zeo.log and uses a custom format for each line. Assuming the example configuration it stored in zeo.config, you can run a server by typing:

runzeo -C zeo.config

A configuration file consists of a <zeo> section and a storage section, where the storage section can use any of the valid ZODB storage types. It may also contain an event log configuration. See ZODB documentation for information on configuring storages.

The zeo section must list the address. All the other keys are optional.

address

The address at which the server should listen. This can be in the form ‘host:port’ to signify a TCP/IP connection or a pathname string to signify a Unix domain socket connection (at least one ‘/’ is required). A hostname may be a DNS name or a dotted IP address. If the hostname is omitted, the platform’s default behavior is used when binding the listening socket (‘’ is passed to socket.bind() as the hostname portion of the address).

read-only

Flag indicating whether the server should operate in read-only mode. Defaults to false. Note that even if the server is operating in writable mode, individual storages may still be read-only. But if the server is in read-only mode, no write operations are allowed, even if the storages are writable. Note that pack() is considered a read-only operation.

invalidation-queue-size

The storage server keeps a queue of the objects modified by the last N transactions, where N == invalidation_queue_size. This queue is used to support client cache verification when a client disconnects for a short period of time.

invalidation-age

The maximum age of a client for which quick-verification invalidations will be provided by iterating over the served storage. This option should only be used if the served storage supports efficient iteration from a starting point near the end of the transaction history (e.g. end of file).

transaction-timeout

The maximum amount of time, in seconds, to wait for a transaction to commit after acquiring the storage lock, specified in seconds. If the transaction takes too long, the client connection will be closed and the transaction aborted.

This defaults to 30 seconds.

client-conflict-resolution

Flag indicating that clients should perform conflict resolution. This option defaults to false.

msgpack

Use msgpack to serialize and de-serialize ZEO protocol messages.

An advantage of using msgpack for ZEO communication is that it’s a tiny bit faster and a ZEO server can support Python 2 or Python 3 clients (but not both).

msgpack can also be enabled by setting the ZEO_MSGPACK environment to a non-empty string.

<zeo>
  address zeo.example.com:8090
  <ssl>
    certificate server_certificate.pem
    key server_certificate_key.pem
  </ssl>
</zeo>

<filestorage>
  path /var/tmp/Data.fs
</filestorage>

<eventlog>
  <logfile>
    path /var/tmp/zeo.log
    format %(asctime)s %(message)s
  </logfile>
</eventlog>

Running the ZEO server as a daemon

In an operational setting, you will want to run the ZEO server as a daemon process that is restarted when it dies. runzeo makes no attempt to implement a well behaved daemon. It expects that functionality to be provided by a wrapper like zdaemon or supervisord.

Rotating log files

runzeo will re-initialize its logging subsystem when it receives a SIGUSR2 signal. If you are using the standard event logger, you should first rename the log file and then send the signal to the server. The server will continue writing to the renamed log file until it receives the signal. After it receives the signal, the server will create a new file with the old name and write to it.

Python API for creating a ZEO client storage

To create a client storage from Python, use the ZEO.client function:

import ZEO
client = ZEO.client(8200)

In the example above, we created a client that connected to a storage listening on port 8200 on local host. The first argument is an address, or list of addresses to connect to. There are many additional options, documented below that should be given as keyword arguments.

Addresses can be:

• A host/port tuple

• An integer, which implies that the host is ‘127.0.0.1’

• A unix domain socket file name.

Options:

cache_size

The cache size in bytes. This defaults to a 20MB.

cache

The ZEO cache to be used. This can be a file name, which will cause a persistent standard persistent ZEO cache to be used and stored in the given name. This can also be an object that implements ZEO.interfaces.ICache.

If not specified, then a non-persistent cache will be used.

blob_dir

The name of a directory to hold/cache blob data downloaded from the server. This must be provided if blobs are to be used. (Of course, the server storage must be configured to use blobs as well.)

shared_blob_dir

A client can use a network files system (or a local directory if the server runs on the same machine) to share a blob directory with the server. This allows downloading of blobs (except via a distributed file system) to be avoided.

blob_cache_size

The size of the blob cache in bytes. IF unset, then blobs will accumulate. If set, then blobs are removed when the total size exceeds this amount. Blobs accessed least recently are removed first.

blob_cache_size_check

The total size of data to be downloaded to trigger blob cache size reduction. The default is 10 (percent). This controls how often to remove blobs from the cache.

ssl

An ssl.SSLContext object used to make SSL connections.

ssl_server_hostname

Host name to use for SSL host name checks.

If using SSL and if host name checking is enabled in the given SSL context then use this as the value to check. If an address is a host/port pair, then this defaults to the host in the address.

read_only

Set to true for a read-only connection.

If false (the default), then request a read/write connection.

This option is ignored if read_only_fallback is set to a true value.

read_only_fallback

Set to true, then prefer a read/write connection, but be willing to use a read-only connection. This defaults to a false value.

If read_only_fallback is set, then read_only is ignored.

server_sync

Flag, false by default, indicating whether the sync method should make a server request. The sync method is called at the start of explicitly begin transactions. Making a server requests assures that any invalidations outstanding at the beginning of a transaction are processed.

Setting this to True is important when application activity is spread over multiple ZEO clients. The classic example of this is when a web browser makes a request to an application server (ZEO client) that makes a change and then makes a request to another application server that depends on the change.

Setting this to True makes transactions a little slower because of the added server round trip. For transactions that don’t otherwise need to access the storage server, the impact can be significant.

wait_timeout

How long to wait for an initial connection, defaulting to 30 seconds. If an initial connection can’t be made within this time limit, then creation of the client storage will fail with a ZEO.Exceptions.ClientDisconnected exception.

After the initial connection, if the client is disconnected:

• In-flight server requests will fail with a ZEO.Exceptions.ClientDisconnected exception.

• New requests will block for up to wait_timeout waiting for a connection to be established before failing with a ZEO.Exceptions.ClientDisconnected exception.

client_label

A short string to display in server logs for an event relating to this client. This can be helpful when debugging.

disconnect_poll

The delay in seconds between attempts to connect to the server, in seconds. Defaults to 1 second.

Configuration strings/files

ZODB databases and storages can be configured using configuration files, or strings (extracted from configuration files). They use the same syntax as the server configuration files described above, but with different sections and options.

An application that used ZODB might configure it’s database using a string like:

<zodb>
   cache-size-bytes 1000MB

   <filestorage>
     path /var/lib/Data.fs
   </filestorage>
</zodb>

In this example, we configured a ZODB database with a object cache size of 1GB. Inside the database, we configured a file storage. The filestorage section provided file-storage parameters. We saw a similar section in the storage-server configuration example in Server configuration.

To configure a client storage, you use a clientstorage section, but first you have to import it’s definition, because ZEO isn’t built into ZODB. Here’s an example:

<zodb>
   cache-size-bytes 1000MB

   %import ZEO

   <clientstorage>
     server 8200
   </clientstorage>
</zodb>

In this example, we defined a client storage that connected to a server on port 8200.

The following settings are supported:

cache-size

The cache size in bytes, KB or MB. This defaults to a 20MB. Optional KB or MB suffixes can (and usually are) used to specify units other than bytes.

cache-path

The file path of a persistent cache file

blob-dir

The name of a directory to hold/cache blob data downloaded from the server. This must be provided if blobs are to be used. (Of course, the server storage must be configured to use blobs as well.)

shared-blob-dir

A client can use a network files system (or a local directory if the server runs on the same machine) to share a blob directory with the server. This allows downloading of blobs (except via a distributed file system) to be avoided.

blob-cache-size

The size of the blob cache in bytes. IF unset, then blobs will accumulate. If set, then blobs are removed when the total size exceeds this amount. Blobs accessed least recently are removed first.

blob-cache-size-check

The total size of data to be downloaded to trigger blob cache size reduction. The default is 10 (percent). This controls how often to remove blobs from the cache.

read-only

Set to true for a read-only connection.

If false (the default), then request a read/write connection.

This option is ignored if read_only_fallback is set to a true value.

read-only-fallback

Set to true, then prefer a read/write connection, but be willing to use a read-only connection. This defaults to a false value.

If read_only_fallback is set, then read_only is ignored.

server-sync

Sets the server_sync option described above.

wait_timeout

How long to wait for an initial connection, defaulting to 30 seconds. If an initial connection can’t be made within this time limit, then creation of the client storage will fail with a ZEO.Exceptions.ClientDisconnected exception.

After the initial connection, if the client is disconnected:

• In-flight server requests will fail with a ZEO.Exceptions.ClientDisconnected exception.

• New requests will block for up to wait_timeout waiting for a connection to be established before failing with a ZEO.Exceptions.ClientDisconnected exception.

client_label

A short string to display in server logs for an event relating to this client. This can be helpful when debugging.

disconnect_poll

The delay in seconds between attempts to connect to the server, in seconds. Defaults to 1 second.

%import ZEO

<clientstorage>
  server zeo.example.com8200
  <ssl>
  </ssl>
</clientstorage>

5.2.1 (unreleased)

• Add support for Python 3.7.

• Switch from msgpack-python to msgpack. Currently a version < 0.6 is required.

• Stop calling the deprecated checkSecure method when creating a ClientStorage. With ZODB 5.2.2 and above, this issued a warning. With older versions, this could issue a log message, but this was considered a misfeature. See issue 134.

5.2.0 (2018-03-28)

• Fixed: The quickstart/ad-hoc/play ZEO server relied on test dependencies. See issue 105.

• Disallow passing strings as addresses to ClientStorage under Windows because string addresses are used for unix-domain sockets, which aren’t supported on Windows. See issue 107.

• Renamed all async attributes to async_ for compatibility with Python 3.7. See issue 104.

• Fixed to work with some changes made in ZODB 5.4.0.

  Client-side updates are incuded for ZODB 5.4.0 or databases that already had zodbpickle.binary OIDs. See issue 113.

• ZEO now uses pickle protocol 3 for both Python 2 and Python 3. (Previously protocol 1 was used for Python 2.) This matches the change in ZODB 5.4.0.

5.1.2 (2018-03-27)

• Fix: ZEO didn’t work with a change in ZODB 5.4.0.

  (Allow zodbpickle.binary to be used in RPC requests, which is necessary for compatibility with ZODB 5.4.0 on Python 2. See issue 107.)

5.1.1 (2017-12-18)

• All classes are new-style classes on Python 2 (they were already new-style on Python 3). This improves performance on PyPy. See issue 86.

• Fixed removing UNIX socket files under Python 2 with ZConfig 3.2.0. See issue 90.

5.1.0 (2017-04-03)

• Added support for serializing ZEO messages using msgpack rather than pickle. This helps pave the way to supporting byteserver, but it also allows ZEO servers to support Python 2 or 3 clients (but not both at the same time) and may provide a small performance improvement.

• Possibly fixed the deprecated and untested zeoctl script.

• Removed zeopasswd, which no longer makes sense given that ZEO authentication was removed, in favor of SSL.

5.0.4 (2016-11-18)

• Fixed: ZEO needed changes to work with recent transaction changes.

  ZEO now works with the latest versions of ZODB and transaction

5.0.3 (2016-11-18)

• Temporarily require non-quite-current versions of ZODB and transaction until we can sort out some recent breakage.

5.0.2 (2016-11-02)

• Provide much better performance on Python 2.

• Provide better error messages when pip tries to install ZEO on an unsupported Python version. See issue 75.

5.0.1 (2016-09-06)

Packaging-related doc fix

5.0.0 (2016-09-06)

This is a major ZEO revision, which replaces the ZEO network protocol implementation.

New features:

• SSL support

• Optional client-side conflict resolution.

• Lots of mostly internal clean ups.

• ClientStorage``server-sync configuration option and server_sync constructor argument to force a server round trip at the beginning of transactions to wait for any outstanding invalidations at the start of the transaction to be delivered.

• Client disconnect errors are now transient errors. When applications retry jobs that raise transient errors, jobs (e.g. web requests) with disconnect errors will be retried. Together with blocking synchronous ZEO server calls for a limited time while disconnected, this change should allow brief disconnections due to server restart to avoid generating client-visible errors (e.g. 500 web responses).

• ClientStorage prefetch method to prefetch oids.

  When oids are prefetched, requests are made at once, but the caller doesn’t block waiting for the results. Rather, then the caller later tries to fetch data for one of the object ids, it’s either delivered right away from the ZEO cache, if the prefetch for the object id has completed, or the caller blocks until the inflight prefetch completes. (No new request is made.)

Dropped features:

• The ZEO authentication protocol.

  This will be replaced by new authentication mechanims leveraging SSL.

• The ZEO monitor server.

• Full cache verification.

• Client suppprt for servers older than ZODB 3.9

• Server support for clients older than ZEO 4.2.0

5.0.0b0 (2016-08-18)

• Added a ClientStorage server-sync configuration option and server_sync constructor argument to force a server round trip at the beginning of transactions to wait for any outstanding invalidations at the start of the transaction to be delivered.

• When creating an ad hoc server, a log file isn’t created by default. You must pass a log option specifying a log file name.

• The ZEO server register method now returns the storage last transaction, allowing the client to avoid an extra round trip during cache verification.

• Client disconnect errors are now transient errors. When applications retry jobs that raise transient errors, jobs (e.g. web requests) with disconnect errors will be retried. Together with blocking synchronous ZEO server calls for a limited time while disconnected, this change should allow brief disconnections due to server restart to avoid generating client-visible errors (e.g. 500 web responses).

• Fixed bugs in using the ZEO 5 client with ZEO 4 servers.

5.0.0a2 (2016-07-30)

• Added the ability to pass credentials when creating client storages.

  This is experimental in that passing credentials will cause connections to an ordinary ZEO server to fail, but it facilitates experimentation with custom ZEO servers. Doing this with custom ZEO clients would have been awkward due to the many levels of composition involved.

  In the future, we expect to support server security plugins that consume credentials for authentication (typically over SSL).

  Note that credentials are opaque to ZEO. They can be any object with a true value. The client mearly passes them to the server, which will someday pass them to a plugin.

5.0.0a1 (2016-07-21)

• Added a ClientStorage prefetch method to prefetch oids.

  When oids are prefetched, requests are made at once, but the caller doesn’t block waiting for the results. Rather, then the caller later tries to fetch data for one of the object ids, it’s either delivered right away from the ZEO cache, if the prefetch for the object id has completed, or the caller blocks until the inflight prefetch completes. (No new request is made.)

• Fixed: SSL clients of servers with signed certs didn’t load default certs and were unable to connect.

5.0.0a0 (2016-07-08)

This is a major ZEO revision, which replaces the ZEO network protocol implementation.

New features:

• SSL support

• Optional client-side conflict resolution.

• Lots of mostly internal clean ups.

Dropped features:

• The ZEO authentication protocol.

  This will be replaced by new authentication mechanims leveraging SSL.

• The ZEO monitor server.

• Full cache verification.

• Client suppprt for servers older than ZODB 3.9

• Server support for clients older than ZEO 4.2.0

4.2.0 (2016-06-15)

• Changed loadBefore to operate more like load behaved, especially with regard to the load lock. This allowes ZEO to work with the upcoming ZODB 5, which used loadbefore rather than load.

  Reimplemented load using loadBefore, thus testing loadBefore extensively via existing tests.

• Other changes to work with ZODB 5 (as well as ZODB 4)

• Fixed: the ZEO cache loadBefore method failed to utilize current data.

• Drop support for Python 2.6 and 3.2.

• Fix AttributeError: ‘ZEOServer’ object has no attribute ‘server’ when StorageServer creation fails.

4.2.0b1 (2015-06-05)

• Add support for PyPy.

4.1.0 (2015-01-06)

• Add support for Python 3.4.

• Added a new ruok client protocol for getting server status on the ZEO port without creating a full-blown client connection and without logging in the server log.

• Log errors on server side even if using multi threaded delay.

4.0.0 (2013-08-18)

• Avoid reading excess random bytes when setting up an auth_digest session.

• Optimize socket address enumeration in ZEO client (avoid non-TCP types).

• Improve Travis CI testing support.

• Assign names to all threads for better runtime debugging.

• Fix “assignment to keyword” error under Py3k in ‘ZEO.scripts.zeoqueue’.

4.0.0b1 (2013-05-20)

• Depend on ZODB >= 4.0.0b2

• Add support for Python 3.2 / 3.3.

4.0.0a1 (2012-11-19)

First (in a long time) separate ZEO release.

Since ZODB 3.10.5:

• Storage servers now emit Serving and Closed events so subscribers can discover addresses when dynamic port assignment (bind to port 0) is used. This could, for example, be used to update address information in a ZooKeeper database.

• Client storages have a method, new_addr, that can be used to change the server address(es). This can be used, for example, to update a dynamically determined server address from information in a ZooKeeper database.