z3c.recipe.scripts 1.0.1

1.0.1 (2010-08-27)

Fixes

• If a section extends another one, it should be able to override it.

• The allowed-eggs-from-site-packages should be looked for in the buildout section if it is not found locally.

1.0.0 (2010-08-23)

(no significant changes)

1.0.0b1 (2010-04-29)

Initial public version.

Script generation

Generating a basic script looks virtually identical to using zc.recipe.egg.

(Note that the find-links and index values are typically not needed; they are included to help make this document run as a test successfully.)

>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = demo
...
... [demo]
... recipe = z3c.recipe.scripts
... eggs = demo<0.3
... find-links = %(server)s
... index = %(server)s/index
... """ % dict(server=link_server))

>>> print system(buildout),
Installing demo.
Getting distribution for 'demo<0.3'.
Got demo 0.2.
Getting distribution for 'demoneeded'.
Got demoneeded 1.2c1.
Generated script '/sample-buildout/bin/demo'.

>>> print system(join(sample_buildout, 'bin', 'demo')),
2 2

Interpreter generation

As with zc.recipe.egg, you can generate an interpreter with the default script recipe shown above by supplying the “interpreter” option. This example will create both an entry point script and an interpreter.

>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = demo
...
... [demo]
... recipe = z3c.recipe.scripts
... eggs = demo<0.3
... find-links = %(server)s
... index = %(server)s/index
... interpreter = py
... """ % dict(server=link_server))

>>> print system(buildout),
Uninstalling demo.
Installing demo.
Generated script '/sample-buildout/bin/demo'.
Generated interpreter '/sample-buildout/bin/py'.

You can also generate an interpreter alone with the interpreter recipe.

>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = py
...
... [py]
... recipe = z3c.recipe.scripts:interpreter
... eggs = demo<0.3
... find-links = %(server)s
... index = %(server)s/index
... """ % dict(server=link_server))

>>> print system(buildout),
Uninstalling demo.
Installing py.
Generated interpreter '/sample-buildout/bin/py'.

In both cases, the bin/py script works by restarting Python after specifying a special path in PYTHONPATH. This example shows the UNIX version; the Windows version actually uses subprocess instead.

>>> cat(sample_buildout, 'bin', 'py') # doctest: +NORMALIZE_WHITESPACE
#!/usr/bin/python2.4 -S
<BLANKLINE>
import os
import sys
<BLANKLINE>
argv = [sys.executable] + sys.argv[1:]
environ = os.environ.copy()
path = '/sample-buildout/parts/py'
if environ.get('PYTHONPATH'):
    path = os.pathsep.join([path, environ['PYTHONPATH']])
environ['PYTHONPATH'] = path
os.execve(sys.executable, argv, environ)

The path is a directory that contains two files: our own site.py and sitecustomize.py. The site.py is modified from the underlying Python’s site.py, and is responsible for setting up our paths. The sitecustomize.py is responsible for running the initialization code provided.

>>> ls(sample_buildout, 'parts', 'py')
-  site.py
-  sitecustomize.py

Here’s an example of using the generated interpreter.

>>> print system(join(sample_buildout, 'bin', 'py') +
...              ' -c "import sys, pprint; pprint.pprint(sys.path[-2:])"')
['/sample-buildout/eggs/demo-0.2-pyN.N.egg',
 '/sample-buildout/eggs/demoneeded-1.2c1-pyN.N.egg']
<BLANKLINE>

Including site-packages and sitecustomize

As introduced above, this recipe supports including site packages. This has some advantages and some serious dangers.

A typical reason to include site-packages is that it is easier to install one or more dependencies in your Python than it is with buildout. Some packages, such as lxml or Python PostgreSQL integration, have dependencies that can be much easier to build and/or install using other mechanisms, such as your operating system’s package manager. By installing some core packages into your Python’s site-packages, this can significantly simplify some application installations.

However, doing this has a significant danger. One of the primary goals of buildout is to provide repeatability. Some packages (one of the better known Python openid packages, for instance) change their behavior depending on what packages are available. If Python curl bindings are available, these may be preferred by the library. If a certain XML package is installed, it may be preferred by the library. These hidden choices may cause small or large behavior differences. The fact that they can be rarely encountered can actually make it worse: you forget that this might be a problem, and debugging the differences can be difficult. If you allow site-packages to be included in your buildout, and the Python you use is not managed precisely by your application (for instance, it is a system Python), you open yourself up to these possibilities. Don’t be unaware of the dangers.

To show off these features, we need to use buildout with a Python executable with some extra paths to show include-site-packages; and one guaranteed to have a sitecustomize module to show exec-sitecustomize. We’ll make one using a test fixture called make_py. The os.environ change below will go into the sitecustomize, and the site_packages_path will be in the Python’s path.

>>> py_path, site_packages_path = make_py(initialization='''\
... import os
... os.environ['zc.buildout'] = 'foo bar baz shazam'
... ''')
>>> print site_packages_path
/executable_buildout/site-packages

Now let’s take a look at include-site-packages. The default is false, so we will set it to true.

>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = py
... executable = %(py_path)s
...
... [py]
... recipe = z3c.recipe.scripts:interpreter
... include-site-packages = true
... eggs = demo<0.3
... find-links = %(server)s
... index = %(server)s/index
... """ % dict(server=link_server, py_path=py_path))

>>> print system(buildout),
Uninstalling py.
Installing py.
Generated interpreter '/sample-buildout/bin/py'.

Now executable_buildout/site-packages is included in sys.path.

>>> print system(join(sample_buildout, 'bin', 'py') +
...              ''' -c "import sys, pprint; pprint.pprint(sys.path)"''')
... # doctest: +ELLIPSIS
['',
 '/sample-buildout/parts/py',
 ...,
 '/sample-buildout/eggs/demo-0.2-pyN.N.egg',
 '/sample-buildout/eggs/demoneeded-1.2c1-pyN.N.egg',
 '/executable_buildout/eggs/setuptools-X-pyN.N.egg',
 '/executable_buildout/site-packages']
<BLANKLINE>

As described above, the allowed-eggs-from-site-packages option lets us control what site-packages eggs zc.buildout will allow to fulfill dependencies. The behavior was described above with an example (and the implementation is tested elsewhere), so we’ll only look at some simple and common use cases here.

Sometimes you may want to allow site-packages to be available but you don’t want your package to depend on it using setup.py. For instance, perhaps you are writing an application, and you want to depend on your system’s packaging of the PostgreSQL code, but the system Python does not use eggs to package it, so you need to manage the two separately. In this case, you might not want to use any eggs from site-packages, but you want it available. In this case, you can use allowed-eggs-from-site-packages with an empty value to keep any egg from being used from site-packages.

Here’s an example. Let’s say we have a Python with demo and demoneeded installed as eggs in the system Python. Normally, they will be used to fulfill dependencies, because allowed-eggs-from-site-packages defaults to the value “*” (allow any package). (We use an empty find-links value to say that buildout may not look elsewhere for the package. We use a different eggs-directory for isolation, so that eggs obtained other parts of the document do not affect this example.)

>>> from zc.buildout.tests import create_sample_sys_install
>>> create_sample_sys_install(site_packages_path)
>>> import zc.buildout.easy_install
>>> zc.buildout.easy_install.clear_index_cache()

>>> write('buildout.cfg',
... '''
... [buildout]
... parts = eggs
... eggs-directory = tmpeggs
... find-links =
...
... [primed_python]
... executable = %(py_path)s
...
... [eggs]
... recipe = z3c.recipe.scripts
... include-site-packages = true
... python = primed_python
... eggs = demoneeded
... ''' % globals())

>>> print system(buildout),
Creating directory '/sample-buildout/tmpeggs'.
Uninstalling py.
Installing eggs.

That succeeds fine, getting demoneeded from the Python site-packages.

However, when allowed-eggs-from-site-packages is an empty value, demoneeded is not allowed to come from site-packages, and the buildout fails.

>>> zc.buildout.easy_install.clear_index_cache()
>>> rmdir('tmpeggs')
>>> write('buildout.cfg',
... '''
... [buildout]
... parts = eggs
... eggs-directory = tmpeggs
... find-links =
...
... [primed_python]
... executable = %(py_path)s
...
... [eggs]
... recipe = z3c.recipe.scripts
... include-site-packages = true
... python = primed_python
... allowed-eggs-from-site-packages =
... eggs = demoneeded
... ''' % globals())
>>> print system(buildout),
Creating directory '/sample-buildout/tmpeggs'.
Uninstalling eggs.
Installing eggs.
Couldn't find index page for 'demoneeded' (maybe misspelled?)
Getting distribution for 'demoneeded'.
While:
  Installing eggs.
  Getting distribution for 'demoneeded'.
Error: Couldn't find a distribution for 'demoneeded'.

The include-sitepackages and allowed-eggs-from-site-packages options both can be obtained from the buildout section if they are not set locally.

Remember that you can provide multiple lines to the allowed-eggs-from-site-packages option, each specifying a whitelist of allowed packages. Globs (* and ?) are allowed.

Next we will use the exec-sitecustomize option. It simply copies Python’s underlying sitecustomize module, if it exists, to the local version. The os.environ change shown above in the make_py call will go into the sitecustomize.

>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = py
... executable = %(py_path)s
...
... [py]
... recipe = z3c.recipe.scripts:interpreter
... exec-sitecustomize = true
... eggs = demo<0.3
... find-links = %(server)s
... index = %(server)s/index
... """ % dict(server=link_server, py_path=py_path))

>>> print system(buildout),
Installing py.
Generated interpreter '/sample-buildout/bin/py'.

>>> cat(sample_buildout, 'parts', 'py', 'sitecustomize.py')
... # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
<BLANKLINE>
# The following is from
# /executable_buildout/parts/py/sitecustomize.py
...
import os
os.environ['zc.buildout'] = 'foo bar baz shazam'

>>> print system(join(sample_buildout, 'bin', 'py') +
...              ''' -c "import os; print os.environ['zc.buildout']"''')
foo bar baz shazam
<BLANKLINE>

It also will be honored in the buildout section if it is not set locally.

Options

We’ll focus now on the remaining options that are different than zc.recipe.egg.

Let’s look at the extends option first.

>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = demo python
...
... [demo]
... recipe = z3c.recipe.scripts
... eggs = demo<0.3
... find-links = %(server)s
... index = %(server)s/index
... initialization =
...     import os
...     os.environ['zc.buildout'] = 'sha boo bop bazoodle'
...
... [python]
... recipe = z3c.recipe.scripts:interpreter
... extends = demo
... initialization =
...     import os
...     os.environ['zc.buildout'] = 'foo bar baz shazam'
... """ % dict(server=link_server))

That makes it easier to specify some initialization for the interpreter that is different than a script, while duplicating other configuration.

Now let’s put it in action.

>>> print system(buildout),
Uninstalling py.
Installing demo.
Generated script '/sample-buildout/bin/demo'.
Installing python.
Generated interpreter '/sample-buildout/bin/python'.

>>> print system(join(sample_buildout, 'bin', 'python') +
...              ' -c "import sys, pprint; pprint.pprint(sys.path[-2:])"')
['/sample-buildout/eggs/demo-0.2-pyN.N.egg',
 '/sample-buildout/eggs/demoneeded-1.2c1-pyN.N.egg']
<BLANKLINE>
>>> print system(join(sample_buildout, 'bin', 'python') +
...              ''' -c "import os; print os.environ['zc.buildout']"'''),
foo bar baz shazam

Note that the parts/py directory has been cleaned up, and parts/python has been created.

>>> ls(sample_buildout, 'parts')
d  buildout
d  demo
d  python

If you want to have initialization that only affects scripts, not the interpreter, you can use script-initialization. Here’s a demonstration.

>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = demo
...
... [demo]
... recipe = z3c.recipe.scripts
... eggs = demo<0.3
... find-links = %(server)s
... index = %(server)s/index
... interpreter = py
... script-initialization =
...     print "Hi from the script"
... """ % dict(server=link_server))

>>> print system(buildout),
Uninstalling python.
Uninstalling demo.
Installing demo.
Generated script '/sample-buildout/bin/demo'.
Generated interpreter '/sample-buildout/bin/py'.

>>> print system(join(sample_buildout, 'bin', 'py') +
...              ''' -c "print 'Hi from the interpreter'"'''),
Hi from the interpreter

>>> print system(join(sample_buildout, 'bin', 'demo')),
Hi from the script
2 2

The last new option is name. This simply changes the name of the interpreter, so that you are not forced to use the name of the section.

>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = interpreter
...
... [interpreter]
... name = python2
... recipe = z3c.recipe.scripts:interpreter
... eggs = demo<0.3
... find-links = %(server)s
... index = %(server)s/index
... """ % dict(server=link_server))

>>> print system(buildout),
Uninstalling demo.
Installing interpreter.
Generated interpreter '/sample-buildout/bin/python2'.

>>> print system(join(sample_buildout, 'bin', 'python2') +
...              ' -c "print 42"')
42
<BLANKLINE>

The other options all identical to zc.recipe.egg.