Recipe for installing Python package distributions as eggs
Project description
The egg-installation recipe installes eggs into a buildout eggs directory. It also generates scripts in a buildout bin directory with egg paths baked into them.
To do
Some way to freeze the egg-versions used. This includes some way to record which versions were selected dynamially and then a way to require that the recorded versions be used in a later run.
Change History
1.0.0b1
Updated to work with zc.buildout 1.0.0b3.
1.0.0a3
Extra path elements to be included in generated scripts can now be set via the extra-paths option.
No longer implicitly generate py_ scripts fo reach egg. There is now an interpreter option to generate a script that, when run without arguments, launches the Python interactive interpreter with the path set based on a parts eggs and extra paths. If this script is run with the name of a Python script and arguments, then the given script is run with the path set.
You can now specify explicit entry points. This is useful for use with packages that don’t declare their own entry points.
Added Windows support.
Now-longer implicitly generate “py_” scripts for each egg. You can now generate a script for launching a Python interpreter or for running scripts based on the eggs defined for an egg part.
You can now specify custom entry points for packages that don’t declare their entry points.
You can now specify extra-paths to be included in generated scripts.
1.0.0a2
Added a custom recipe for building custom eggs using custom distrutils build_ext arguments.
1.0.0a1
Initial public version
Detailed Documentation
Installation of distributions as eggs
The zc.recipe.egg recipe can be used to install various types if distutils distributions as eggs. It takes a number of options:
- eggs
A list of eggs to install given as one ore more setuptools requirement strings. Each string must be given on a separate line.
- find-links
A list of URLs, files, or directories to search for distributions.
- index
The URL of an index server, or almost any other valid URL. :)
If not specified, the Python Package Index, http://cheeseshop.python.org/pypi, is used. You can specify an alternate index with this option. If you use the links option and if the links point to the needed distributions, then the index can be anything and will be largely ignored. In the examples, here, we’ll just point to an empty directory on our link server. This will make our examples run a little bit faster.
- python
The name of a section to get the Python executable from. If not specified, then the buildout python option is used. The Python executable is found in the executable option of the named section.
- entry-points
A list of entry-point identifiers of the form name=module#attrs, name is a script name, module is a module name, and a attrs is a (possibly dotted) name of an object wihin the module. This option is useful when working with distributions that don’t declare entry points, such as distributions not written to work with setuptools.
- scripts
Control which scripts are generated. The value should be a list of zero or more tokens. Each token is either a name, or a name followed by an ‘=’ and a new name. Only the named scripts are generated. If no tokens are given, then script generation is disabled. If the option isn’t given at all, then all scripts defined by the named eggs will be generated.
- interpreter
The name of a script to generate that allows access to a Python interpreter that has the path set based on the eggs installed.
- extra-paths
Extra paths to include in a generates script.
We have a link server that has a number of distributions:
>>> print get(link_server), <html><body> <a href="demo-0.1-py2.3.egg">demo-0.1-py2.3.egg</a><br> <a href="demo-0.2-py2.3.egg">demo-0.2-py2.3.egg</a><br> <a href="demo-0.3-py2.3.egg">demo-0.3-py2.3.egg</a><br> <a href="demoneeded-1.0.zip">demoneeded-1.0.zip</a><br> <a href="demoneeded-1.1.zip">demoneeded-1.1.zip</a><br> <a href="extdemo-1.4.zip">extdemo-1.4.zip</a><br> <a href="index/">index/</a><br> <a href="other-1.0-py2.3.egg">other-1.0-py2.3.egg</a><br> </body></html>
We have a sample buildout. Let’s update it’s configuration file to install the demo package.
>>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... parts = demo ... ... [demo] ... recipe = zc.recipe.egg ... eggs = demo<0.3 ... find-links = %(server)s ... index = %(server)s/index ... """ % dict(server=link_server))
In this example, we limited ourselves to revisions before 0.3. We also specified where to find distributions using the find-links option.
Let’s run the buildout:
>>> import os >>> os.chdir(sample_buildout) >>> buildout = os.path.join(sample_buildout, 'bin', 'buildout') >>> print system(buildout), buildout: Installing demo zc.buildout.easy_install: Getting new distribution for demo<0.3 zc.buildout.easy_install: Got demo 0.2 zc.buildout.easy_install: Getting new distribution for demoneeded zc.buildout.easy_install: Got demoneeded 1.1
Now, if we look at the buildout eggs directory:
>>> ls(sample_buildout, 'eggs') - demo-0.2-py2.3.egg - demoneeded-1.1-py2.3.egg - setuptools-0.6-py2.3.egg - zc.buildout-1.0-py2.3.egg
We see that we got an egg for demo that met the requirement, as well as the egg for demoneeded, wich demo requires. (We also see an egg link for the recipe. This egg link was actually created as part of the sample buildout setup. Normally, when using the recipe, you’ll get a regular egg installation.)
Script generation
The demo egg also defined a script and we see that the script was installed as well:
>>> ls(sample_buildout, 'bin') - buildout - demo
Here, in addition to the buildout script, we see the demo script, demo.
Let’s add an interpreter option:
>>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... parts = demo ... ... [demo] ... recipe = zc.recipe.egg ... eggs = demo<0.3 ... find-links = %(server)s ... index = %(server)s/index ... interpreter = py-demo ... """ % dict(server=link_server))>>> print system(buildout), buildout: Uninstalling demo buildout: Installing demo
Now we also get a py-demo script for giving us a Python prompt with the path for demo and any eggs it depends on included in sys.path. This is useful for debugging and testing.
>>> ls(sample_buildout, 'bin') - buildout - demo - py-demo
If we run the demo script, it prints out some minimal data:
>>> print system(os.path.join(sample_buildout, 'bin', 'demo')), 2 1
The value it prints out happens to be some values defined in the modules installed.
We can also run the py-demo script. Here we’ll just print out the bits if the path added to reflect the eggs:
>>> print system(os.path.join(sample_buildout, 'bin', 'py-demo'), ... """import os, sys ... for p in sys.path: ... if 'demo' in p: ... print os.path.basename(p) ... ... """).replace('>>> ', '').replace('... ', ''), ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE demo-0.2-py2.4.egg demoneeded-1.1-py2.4.egg
The recipe gets the most recent distribution that satisfies the specification. For example, We remove the restriction on demo:
>>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... parts = demo ... ... [demo] ... recipe = zc.recipe.egg ... find-links = %(server)s ... index = %(server)s/index ... """ % dict(server=link_server))>>> print system(buildout), buildout: Uninstalling demo buildout: Installing demo zc.buildout.easy_install: Getting new distribution for demo zc.buildout.easy_install: Got demo 0.3
Then we’ll get a new demo egg:
>>> ls(sample_buildout, 'eggs') - demo-0.2-py2.3.egg - demo-0.3-py2.3.egg - demoneeded-1.1-py2.3.egg - setuptools-0.6-py2.4.egg - zc.buildout-1.0-py2.4.egg
Note that we removed the eggs option, and the eggs defaulted to the part name.
The script is updated too:
>>> print system(os.path.join(sample_buildout, 'bin', 'demo')), 3 1
Controlling script generation
You can control which scripts get generated using the scripts option. For example, to suppress scripts, use the scripts option without any arguments:
>>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... parts = demo ... ... [demo] ... recipe = zc.recipe.egg ... find-links = %(server)s ... index = %(server)s/index ... scripts = ... """ % dict(server=link_server))>>> print system(buildout), buildout: Uninstalling demo buildout: Installing demo>>> ls(sample_buildout, 'bin') - buildout
You can also control the name used for scripts:
>>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... parts = demo ... ... [demo] ... recipe = zc.recipe.egg ... find-links = %(server)s ... index = %(server)s/index ... scripts = demo=foo ... """ % dict(server=link_server))>>> print system(buildout), buildout: Uninstalling demo buildout: Installing demo>>> ls(sample_buildout, 'bin') - buildout - foo
Specifying extra script paths
If we need to include extra paths in a script, we can use the extra-paths option:
>>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... parts = demo ... ... [demo] ... recipe = zc.recipe.egg ... find-links = %(server)s ... index = %(server)s/index ... scripts = demo=foo ... extra-paths = ... /foo/bar ... /spam/eggs ... """ % dict(server=link_server))>>> print system(buildout), buildout: Uninstalling demo buildout: Installing demo
Let’s look at the script that was generated:
>>> cat(sample_buildout, 'bin', 'foo') # doctest: +NORMALIZE_WHITESPACE #!/usr/local/bin/python2.4 <BLANKLINE> import sys sys.path[0:0] = [ '/sample-buildout/eggs/demo-0.3-py2.4.egg', '/sample-buildout/eggs/demoneeded-1.1-py2.4.egg', '/foo/bar', '/spam/eggs', ] <BLANKLINE> import eggrecipedemo <BLANKLINE> if __name__ == '__main__': eggrecipedemo.main()
Specifying entry points
Scripts can be generated for entry points declared explcitly. We can declate entry points using the entry-points option:
>>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... parts = demo ... ... [demo] ... recipe = zc.recipe.egg ... find-links = %(server)s ... index = %(server)s/index ... extra-paths = ... /foo/bar ... /spam/eggs ... entry-points = alt=eggrecipedemo:alt other=foo.bar:a.b.c ... """ % dict(server=link_server))>>> print system(buildout), buildout: Uninstalling demo buildout: Installing demo>>> ls(sample_buildout, 'bin') - alt - buildout - demo - other>>> cat(sample_buildout, 'bin', 'other') #!/usr/local/bin/python2.4 <BLANKLINE> import sys sys.path[0:0] = [ '/sample-buildout/eggs/demo-0.3-py2.4.egg', '/sample-buildout/eggs/demoneeded-1.1-py2.4.egg', '/foo/bar', '/spam/eggs', ] <BLANKLINE> import foo.bar <BLANKLINE> if __name__ == '__main__': foo.bar.a.b.c()
Offline mode
If the buildout offline option is set to “true”, then no attempt will be made to contact an index server:
>>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... parts = demo ... offline = true ... ... [demo] ... recipe = zc.recipe.egg ... index = eek! ... scripts = demo=foo ... """ % dict(server=link_server))>>> print system(buildout), buildout: Uninstalling demo buildout: Installing demo
Controlling which Python to use
The following assumes that you have Python 2.3 installed.
We can specify the python to use by specifying the name of a section to read the Python executable from. The default is the section defined by the python buildout option.
We have a link server:
>>> print get(link_server), <html><body> <a href="demo-0.1-py2.3.egg">demo-0.1-py2.3.egg</a><br> <a href="demo-0.2-py2.3.egg">demo-0.2-py2.3.egg</a><br> <a href="demo-0.3-py2.3.egg">demo-0.3-py2.3.egg</a><br> <a href="demoneeded-1.0.zip">demoneeded-1.0.zip</a><br> <a href="demoneeded-1.1.zip">demoneeded-1.1.zip</a><br> <a href="extdemo-1.4.zip">extdemo-1.4.zip</a><br> <a href="index/">index/</a><br> <a href="other-1.0-py2.3.egg">other-1.0-py2.3.egg</a><br> </body></html>
We have a sample buildout. Let’s update it’s configuration file to install the demo package using Python 2.3.
>>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... parts = demo ... eggs-directory = eggs ... index = http://www.python.org/pypi/ ... ... [python2.3] ... executable = %(python23)s ... ... [demo] ... recipe = zc.recipe.egg ... eggs = demo <0.3 ... find-links = %(server)s ... python = python2.3 ... interpreter = py-demo ... """ % dict(server=link_server, python23=python2_3_executable))
Now, if we run the buildout:
>>> import os >>> os.chdir(sample_buildout) >>> buildout = os.path.join(sample_buildout, 'bin', 'buildout') >>> print system(buildout), buildout: Installing demo zc.buildout.easy_install: Getting new distribution for demo<0.3 zc.buildout.easy_install: Got demo 0.2 zc.buildout.easy_install: Getting new distribution for demoneeded zc.buildout.easy_install: Getting new distribution for setuptools zc.buildout.easy_install: Got setuptools 0.6 zc.buildout.easy_install: Got demoneeded 1.1
we’ll get the Python 2.3 eggs for demo and demoneeded:
>>> ls(sample_buildout, 'eggs') - demo-0.2-py2.3.egg - demoneeded-1.1-py2.3.egg - setuptools-0.6-py2.3.egg - setuptools-0.6-py2.4.egg - zc.buildout-1.0-py2.4.egg
And the generated scripts invoke Python 2.3:
>>> import sys >>> if sys.platform == 'win32': ... script_name = 'demo-script.py' ... else: ... script_name = 'demo' >>> f = open(os.path.join(sample_buildout, 'bin', script_name)) >>> f.readline().strip() == '#!' + python2_3_executable True >>> print f.read(), # doctest: +NORMALIZE_WHITESPACE <BLANKLINE> import sys sys.path[0:0] = [ '/sample-buildout/eggs/demo-0.2-py2.3.egg', '/sample-buildout/eggs/demoneeded-1.1-py2.3.egg', ] <BLANKLINE> import eggrecipedemo <BLANKLINE> if __name__ == '__main__': eggrecipedemo.main()>>> if sys.platform == 'win32': ... f = open(os.path.join(sample_buildout, 'bin', 'py-demo-script.py')) ... else: ... f = open(os.path.join(sample_buildout, 'bin', 'py-demo')) >>> f.readline().strip() == '#!' + python2_3_executable True >>> print f.read(), # doctest: +NORMALIZE_WHITESPACE import sys <BLANKLINE> sys.path[0:0] = [ '/sample-buildout/eggs/demo-0.2-py2.3.egg', '/sample-buildout/eggs/demoneeded-1.1-py2.3.egg', ] <BLANKLINE> _interactive = True if len(sys.argv) > 1: import getopt _options, _args = getopt.getopt(sys.argv[1:], 'ic:') _interactive = False for (_opt, _val) in _options: if _opt == '-i': _interactive = True elif _opt == '-c': exec _val <BLANKLINE> if _args: sys.argv[:] = _args execfile(sys.argv[0]) <BLANKLINE> if _interactive: import code code.interact(banner="", local=globals())>>> f.close()
Custon eggs
Sometimes, It’s necessary to provide extra control over how an egg is created. This is commonly true for eggs with extension modules that need to access libraries or include files.
The zc.recipe.egg:custom recipe can be used to define an egg with custom build parameters. The currently defined parameters are:
- include-dirs
A new-line separated list of directories to search for include files.
- library-dirs
A new-line separated list of directories to search for libraries to link with.
- rpath
A new-line separated list of directories to search for dynamic libraries at run time.
In addition, the following options can be used to specify the egg:
- egg
An eggs to install given as a setuptools requirement string. This defaults to the part name.
- find-links
A list of URLs, files, or directories to search for distributions.
- index
The URL of an index server, or almost any other valid URL. :)
If not specified, the Python Package Index, http://cheeseshop.python.org/pypi, is used. You can specify an alternate index with this option. If you use the links option and if the links point to the needed distributions, then the index can be anything and will be largely ignored. In the examples, here, we’ll just point to an empty directory on our link server. This will make our examples run a little bit faster.
- python
The name of a section to get the Python executable from. If not specified, then the buildout python option is used. The Python executable is found in the executable option of the named section.
To illustrate this, we’ll define a buildout that builds an egg for a package that has a simple extension module:
#include <Python.h> #include <extdemo.h> static PyMethodDef methods[] = {}; PyMODINIT_FUNC initextdemo(void) { PyObject *d; d = Py_InitModule3("extdemo", methods, ""); PyDict_SetItemString(d, "val", PyInt_FromLong(EXTDEMO)); }
The extension depends on a system-dependnt include file, extdemo.h, that defines a constant, EXTDEMO, that is exposed by the extension.
The extension module is available as a source distribution, extdemo-1.4.tar.gz, on a distribution server.
We have a sample buildout that we’ll add an include directory to with the necessary include file:
>>> mkdir(sample_buildout, 'include') >>> import os >>> open(os.path.join(sample_buildout, 'include', 'extdemo.h'), 'w').write( ... "#define EXTDEMO 42\n")
We’ll also update the buildout configuration file to define a part for the egg:
>>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... parts = extdemo ... ... [extdemo] ... recipe = zc.recipe.egg:custom ... find-links = %(server)s ... index = %(server)s/index ... include-dirs = include ... """ % dict(server=link_server))>>> os.chdir(sample_buildout) >>> buildout = os.path.join(sample_buildout, 'bin', 'buildout')>>> print system(buildout), buildout: Installing extdemo zip_safe flag not set; analyzing archive contents...
We got the zip_safe warning because the source distribution we used wasn’t setuptools based and thus didn’t set the option.
The egg is created in the develop-eggs directory not the eggs directory because it depends on buildout-specific parameters and the eggs directory can be shared across multiple buildouts.
>>> ls(sample_buildout, 'develop-eggs') d extdemo-1.4-py2.4-unix-i686.egg - zc.recipe.egg.egg-link
Note that no scripts or dependencies are installed. To install dependencies or scripts for a custom egg, define another part and use the zc.recipe.egg recipe, listing the custom egg as one of the eggs to be installed. The zc.recipe.egg recipe will use the installed egg.
Egg Recipe API for other Recipes
It is common for recipes to accept a collection of egg specifications and generate scripts based on the resulting working sets. The egg recipe provides an API that other recipes can use.
A recipe can reuse the egg recipe, supporting the eggs, find-links, index, extra-paths, and python options. This is done by creating an egg recipe instance in a recipes’s contructor. In the recipe’s install script, the egg-recipe instance’s working_set method is used to collect the requested eggs and working set.
To illustrate, we create a sample recipe that is a very thin layer around the egg recipe:
>>> mkdir(sample_buildout, 'sample') >>> write(sample_buildout, 'sample', 'sample.py', ... """ ... import logging, os ... import zc.recipe.egg ... ... class Sample: ... ... def __init__(self, buildout, name, options): ... self.egg = zc.recipe.egg.Egg(buildout, name, options) ... self.name = name ... self.options = options ... ... def install(self): ... extras = self.options['extras'].split() ... requirements, ws = self.egg.working_set(extras) ... print 'Part:', self.name ... print 'Egg requirements:' ... for r in requirements: ... print r ... print 'Working set:' ... for d in ws: ... print d ... print 'extra paths:', self.egg.extra_paths ... """)
Here we instantiated the egg recipe in the constructor, saving it in an attribute. This also initialized the options dictionary.
In our install method, we called the working_set method on the instance we saved. The working_set method takes an optional sequence of extra requirements to be included in the working set.
>>> write(sample_buildout, 'sample', 'setup.py', ... """ ... from setuptools import setup ... ... setup( ... name = "sample", ... entry_points = {'zc.buildout': ['default = sample:Sample']}, ... install_requires = 'zc.recipe.egg', ... ) ... """)>>> write(sample_buildout, 'sample', 'README.txt', " ")>>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = sample ... parts = sample-part ... ... [sample-part] ... recipe = sample ... eggs = demo<0.3 ... find-links = %(server)s ... index = %(server)sindex ... extras = other ... """ % dict(server=link_server))>>> import os >>> os.chdir(sample_buildout) >>> buildout = os.path.join(sample_buildout, 'bin', 'buildout') >>> print system(buildout + ' -q'), Part: sample-part Egg requirements: demo<0.3 Working set: demo 0.2 other 1.0 demoneeded 1.1 extra paths: []
We can see that the options were augmented with additionl data computed by the egg recipe by looking at .installed.cfg:
>>> cat(sample_buildout, '.installed.cfg') [buildout] parts = sample-part <BLANKLINE> [sample-part] __buildout_installed__ = __buildout_signature__ = sample-6aWMvV2EJ9Ijq+bR8ugArQ== zc.recipe.egg-cAsnudgkduAa/Fd+WJIM6Q== setuptools-0.6-py2.4.egg zc.buildout-+rYeCcmFuD1K/aB77XTj5A== _b = /tmp/tmpb7kP9bsample-buildout/bin _d = /tmp/tmpb7kP9bsample-buildout/develop-eggs _e = /tmp/tmpb7kP9bsample-buildout/eggs eggs = demo<0.3 executable = /usr/local/bin/python2.3 extras = other find-links = http://localhost:27071/ index = http://localhost:27071/index recipe = sample
If we use the extra-paths option:
>>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... develop = sample ... parts = sample-part ... ... [sample-part] ... recipe = sample ... eggs = demo<0.3 ... find-links = %(server)s ... index = %(server)sindex ... extras = other ... extra-paths = /foo/bar ... /spam/eggs ... """ % dict(server=link_server))
Then we’ll see that reflected in the extra_paths attribute in the egg recipe instance:
>>> print system(buildout + ' -q'), Part: sample-part Egg requirements: demo<0.3 Working set: demo 0.2 other 1.0 demoneeded 1.1 extra paths: ['/foo/bar', '/spam/eggs']
Download
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.