ulif.openoffice 0.3

What ulif.openoffice is

ulif.openoffice is a Python package to support document transformations using OpenOffice.org (OOo).

It provides components to ask a running OOo-server for document conversions from office-type documents like .doc or .odt to HTML or PDF. Using ulif.openoffice you can trigger such conversions via commandline or via a Python-API that works also with Python versions without any PyUNO support.

Furthermore, it provides a caching server that caches all documents once converted and delivers them in case a document is requested again. Depending on your needs this can speed-up things by factor 10 or more.

Sources

ulif.openoffice is hosted on:

http://pypi.python.org/pypi/ulif.openoffice

where you can get latest released versions.

The subversion repository of the sources is:

https://svn.gnufix.de/repos/main/ulif.openoffice/

Requirements

ulif.openoffice requires some PyUNO-capable Python somewhere to do the actual conversions. It also provides a client-API for Python code that does not provide that support. Current Debian-based distributions normally offer a package for PyUNO support.

ulif.openoffice is tested on Debian-based systems, most notably Ubuntu, and won’t work on Windows.

The package is designed for server-based deployments. While the OOo-server is running, you cannot use the office-suite on your desktop (at least at time of writing this). This is a limitation of OOo itself.

Overview

ulif.openoffice mainly provides three different components:

• An oooctl server that runs in background, starts a local OOo-server and monitors its status. If the OOo server process dies, it is restarted by oooctl.

• A pyunoserver, which is a TCP-server that implements an own protocol to listen for conversion requests. When a valid request arrives, it tries to contact a local OOo server to do the conversion.

  The pyunoserver also runs a cache manager that caches already converted documents and delivers them in case the conversioned version already exists.

  This component needs access to the PyUNO library.

• A client library to talk to the PyUNO server. This component does not require PyUNO.

The three components play together roughly as shown in the following figure:

The blue lines show the way from a source document (in .doc format) to the OpenOffice.org server while the red lines show the way back of the converted document (PDF).

Use of client-API, oooctl server and cache is optional.

All this currently happens on the same machine. There are plans for support of multi-machine scenarios with distributed servers and load-balancing features.

Prerequisites

There are, unfortunately, zillions of possibilities why you cannot start OpenOffice.org as in background on a system.

The scripts in here were tested with Ubuntu and work.

It is mandatory, that the system user running oooctl is a regular user with at least a home directory. OpenOffice.org relies on that directory to store information even in headless mode.

Recent OpenOffice.org versions require no X-server for running.

If you want to use a Ubuntu (or Debian) prepared install of OOo, you must make sure, that you apt-get-installed the following packages:

• openoffice.org-headless (for Ubuntu < 9.04, not needed for newer)

• openoffice.org-java-common

additionally to the usual OOo packages, i.e.:

• openoffice.org (at least for Ubuntu >= 9.04)

• msttcorefonts

The latter is optional but needed to have the most common fonts used with OpenOffice.org documents available. Without the correct fonts installed, results of document transforms might be poor.

Then, you need at least one Python version, which supports:

$ python -c "import uno"

without raising any exceptions.

On newer Ubuntu versions you can install:

* ``python-uno`` (if available)

The clients and other software apart from the oooctl-server and the pyuno-server can be run with a different Python version.

If you successfully installed this package on a different system, we’d be glad to hear from you, especially, if you could tell, what system-packages you used.

Building

$ /usr/bin/python -c "import uno"

$ python bootstrap/bootstrap.py

$ ./bin/buildout

Using the scripts

There are four main components that come with ulif.openoffice:

• an oooctl-server that starts OpenOffice.org in background.

• a pyuno-server that listens for requests to convert docs. This server depends on a running oooctl-server.

• a client component that can be accessed via API and can talk to the pyuno-server. This way you can convert docs from Python and the Python version has not to provide the uno lib.

• a converter script (also in ./bin), you can use on the commandline. It depends on a running oooctl server and can convert docs to .txt, .html and .pdf format. It is merely a little test programme that was used during development, but you might have some use for it.

You can start the oooctl-server with:

$ ./bin/oooctl start

Do:

$ ./bin/oooctl --help

to see all options.

You can stop the daemon with:

$ ./bin/oooctl stop

The same applies to the pyuno-server:

$ ./bin/pyunoctl start
$ ./bin/pyunoctl --help
$ ./bin/pyunoctl stop

do what you think they do.

The converter script can be called like this:

$ ./bin/convert sourcefile.doc

to create a sourcefile.txt conversion.

Do:

$ ./bin/convert --pdf sourcefile.doc

to create a PDF of sourefile.doc, and:

$ ./bin/convert --html sourcefile.doc

to create an HTML version of sourcefile.doc.

For the client API see the .txt files in the source.

Examples

oooctl – the OOo daemon

We can start an OpenOffice.org daemon using the oooctl script. This daemon starts an already installed OpenOffice.org instance as server (without GUI, so it is usable on servers).

The oooctl script is defined in setup.py to be installed as a console script, so if you install ulif.openoffice with easy_install or setup.py, an executable script will be installed in your local bin/ directory.

Here we ‘fake’ this install by using buildout, which will install the script in our test environment.

To do so we create a buildout.cfg file:

>>> write('buildout.cfg',
... '''
... [buildout]
... parts = openoffice-ctl
... offline = true
...
... [unopython]
... executable = /usr/bin/python
...
... [openoffice-ctl]
... recipe = zc.recipe.egg
... eggs = ulif.openoffice
... python = unopython
... ''')

Now we can run buildout to install our script (and other scripts, described below, as well):

>>> print system(join('bin', 'buildout'))
Installing openoffice-ctl.
Generated script '.../bin/pyunoctl'.
Generated script '.../bin/convert'.
Generated script '.../bin/oooctl'.
<BLANKLINE>

The script provides help with the -h switch:

>>> print system(join('bin', 'oooctl') + ' -h')
Usage: oooctl [options] start|fg|stop|restart|status
...

The main actions are to call the script with one of the:

start|fg|stop|restart|status

commands as argument, where fg means: start in foreground. This can be handy, if you want the process to be monitored by third-party tools like some supervisor daemon or similar. In that case the process will not detach from the invoking shell on startup.

We set the oooctl path as a var for further use:

>>> oooctl_bin = join('bin', 'oooctl')

pyunoctl – a conversion daemon

This script starts a server in background that allows conversion of documents using the pyUNO API. It requires a running OO.org server in background (see above).

Currently conversion from all OOo readable formats (.doc, .odt, .txt, …) to HTML and PDF-A is supported. This means, if you can load a document with OpenOffice.org, then this daemon can convert it to HTML or PDF-A.

The conversion daemon starts a server in background (unless you specify fg as startmode, which will keep the server attached to the invoking shell) which listens for conversion requests on a TCP port. It then calls OpenOffice.org via the pyUNO-API to perform the conversion and responses with the path of the generated doc (or an error message).

The conversion server is a multithreaded asynchronous TCP daemon. So, several requests can be served at the same time.

The script provides help with the -h switch:

>>> print system(join('bin', 'pyunoctl') + ' -h')
Usage: pyunoctl [options] start|fg|stop|restart|status
...

>>> import os

Before we can really use the daemon, we have to fire up the OOo daemon:

>>> print system(join('bin', 'oooctl') + ' --stdout=/tmp/output start')
starting OpenOffice.org server, going into background...
started with pid ...
<BLANKLINE>

Now, we start the pyuno daemon:

>>> print system(join('bin', 'pyunoctl') + ' --stdout=/tmp/out start')
starting pyUNO conversion server, going into background...
started with pid ...
<BLANKLINE>

CONVERT_PDF
PATH=<path-to-source-document>

CONVERT_HTML
PATH=<path-to-source-document>

pyunoctl – RESTful mode

This script starts a server in background that allows conversion of documents using the pyUNO API. It requires a running OO.org server in background (see above).

Apart from usage in standard raw mode, pyunoctl can also be started as a RESTful HTTP daemon. This enables usage from remote, as all communication is done using the HTTP protocol (including sending and receiving files).

The RESTful HTTP mode can be enabled by setting the:

--mode=rest

option of pyunoctl.

We start pyunoctl in RESTful mode. The OOo daemon was already started before.

>>> print system(join('bin', 'pyunoctl') + ' --stdout=/tmp/out '
...              + '--mode=rest start')
startung RESTful HTTP server, going into background...
started with pid ...
<BLANKLINE>

We send a simple test request, that should give us a status:

>>> import httplib
>>> conn = httplib.HTTPConnection('localhost', 2009)
>>> conn.request('GET', '/TEST')
>>> r1 = conn.getresponse()
>>> r1.status, r1.reason
(200, 'OK')

>>> print r1.read()
Server: ulif.openoffice.RESTfulHTTPServer/<VERSION> Python/2.5.2
Date: ...
Content-Length:: ...
<BLANKLINE>
ulif.openoffice.RESTful.HTTPServer <VERSION>

We GET documents from the server by asking for an existing MD5sum. The MD5 sum of a document is also its resource name on the server. If a document does not exist, we get a 404 error:

>>> conn.request('GET', '/non-existing-url')
>>> r1 = conn.getresponse()
>>> r1.status, r1.reason
(404, 'Not Found')

>>> conn.close()

We ask for conversion (creating a resource), simply by POSTing a document. As creating POST request is a bit more complex, we use utility functions from the util module:

>>> from ulif.openoffice.util import encode_multipart_formdata
>>> fields = []
>>> files = [('document', 'simpledoc1.doc',
...           open(testdoc_path, 'rb').read())]
>>> content_type, body = encode_multipart_formdata(fields, files)
>>> headers = {
...    'User-Agent': 'Test-Agent',
...    'Content-Type': content_type
... }

The content type we use has to be multipart/form-data (instead of x-application/urlencoded):

>>> content_type
'multipart/form-data; boundary=---...'

Actually, we can trigger the POST request:

>>> conn = httplib.HTTPConnection('localhost', 2009)
>>> conn.request('POST', '/', body, headers)
>>> r1 = conn.getresponse()
>>> r1.status, r1.reason
(200, 'OK')

>>> print r1.read()
Client: ('127.0.0.1', ...)
Path: /
Form data:
    Uploaded document (name=simpledoc1.doc; 64512 bytes)

Shut down the pyuno daemon:

>>> print system(join('bin', 'pyunoctl') + ' stop')
stopping pid ... done.
<BLANKLINE>

Shut down the oooctl daemon:

>>> print system(join('bin', 'oooctl') + ' stop')
stopping pid ... done.
<BLANKLINE>

Clean up:

>>> os.close(tmp_fd)
>>> os.unlink(tmp_path)

0.3 (2010-11-17)

• Added option to disable caching completely: set --cache-dir to empty string to disable caching [Thanks to Adama Groszer for patches!]

• Removed unwanted output when running in foreground mode.

• Cachemanager now supports listing all sources contained in cache dir.

• Fixed bug in cachemanager: under rare circumstances (two different input files with same MD5 hash digest and identical file stats were considered to be identical by the cachemanager and thus led to inconsistencies in cache). We now check thoroughly whether two such files differ.

• Lots of test fixes [Thanks to Adam Groszer for patches!]

0.2.1 (2010-06-13)

• Fixed fix to cope with pyuno monkey-patching standard __import__ function. More recent pyuno versions do not do that kind of stuff any more (which is an improvement).

• Fixed foreground start of `oooctl` server. It didn’t work correctly with more recent OpenOffice.org/pyuno installs. You now don’t have to press CTRL-C two times anymore when trying to stop a oooctl server running in foreground.

0.2 (2010-05-20)

• Added license and copyright file to comply with policy of major Linux distributors.

• Added sphinx docs.

• Fixed wrong result path when returning cached HTML results.

• Added mode fg for oooctl. Using oooctl fg one can start oooctl in foreground now.

• Added mode fg for pyunoctl. Using pyunoctl fg one can start pyunoctl in foreground now.

• Added state check for oooctl: when OpenOffice.org server is down during runtime it is restarted automatically. The check happens every second.

• Use standard lib doctest instead of zope.testing.doctest.

• Changed PDF creation: by default now normal PDF (and not PDF/A) is created when converting to PDF. This is due to an endianess bug in many recent OpenOffice.org binaries running on 64-bit platforms.

0.1 (2010-03-02)

• Initial implementation.