p01.build 0.6.0

Sample

As an example, consider a web application called MyServer:

• p01.cdn - cdn resources like css and javascript files

• p01.core - a package providing your application model

• p01.web - a web front end for the p01.core package and using the p01.cdn resources

Using p01.build you will be able to manage the lifecycle of these python packages and any number of deployment configurations coherently:

• Allow you to define a project, which is a collection of interdependent eggs that generally get released together.

• Automatically deploy new egg releases of each package when necessary.

• Upload new eggs to a private egg repository (see mpypi).

• Generate versioned buildout configuration files that combine the eggs properly.

• Upload the buildout configuration files to a private configuration server. (mypypi has built in support such uploads)

• Upload dependent buildout configuration files to a private configuration server (by checking the extends= chain, mypypi support such uploads)

Installation

Install the deploy script with easy_install:

$ easy_install p01.build

You can also install a developer version of p01.build

Checkout the code:

$ svn checkout svn://svn.zope.org/repos/main/p01.build/trunk p01.build
$ cd p01.build

Run the bootstrap script and buildout:

$ python bootstrap.py
$ ./bin/buildout

Release

Once you have working developer version available, you should be able to run the build-package script. (Found in ./bin/build-package with developer installations):

$ build-package
Usage: build-package [options]

Options:
  -h, --help            show this help message and exit
  -c FILE, --config-file=FILE
                        The file containing the configuration of the project.
  -q, --quiet           When specified, no messages are displayed.
  -v, --verbose         When specified, debug information is created.
  -d, --use-defaults    When specified, no user input is required and the defaults are used.
  -o, --offline-mode    When set, no server commands are executed.
  -n, --next-version    When set, the system guesses the next version to
                        generate.
  -b BRANCH, --use-branch=BRANCH
                        When specified, this branch will be always used.
  -i BRANCHES --independent-branches=BRANCH1 BRANCH2,
                        When specified, the system guesses the next version from all this branches. This is important to set if
                        you release from different branches. It will prevent
                        that a package get used which was released from
                        another branch !!!
  --no-upload           When set, the generated configuration files are not
                        uploaded.
  --no-branch-update    When set, the branch is not updated with a new version
                        after a release is created.
  -s PATH, --storage-path=PATH
                        Store the generated files in that folder and not in
                        the global root directory. This is just for provide
                        a clean setup and prevent to put all the generated config files

Getting Started

Assumptions; First we need to start with some assumptions that p01.build has about the way packages are laid out. When developing multiple packages in tandem, it often makes sense to have your subversion repository laid out like so:

SVNROOT/MyServer/packages/
    branches/
        Branch-0.x/
            ...
        Branch-1.x/
            p01.cdn/
            p01.core/
            p01.web/
    tags/
        p01.cdn-0.5.0/
        p01.core-0.5.0/
        p01.core-0.5.1/
        p01.web-0.5.0/
        p01.web-0.5.1/
        p01.web-0.5.2/
        ...
    trunk/
        p01.cdn/
        p01.core/
        p01.web/

The important thing to note is that each package does not have its own branches/ tags/ trunk/ directories, but rather there is just one set of the entire “project.”

Project setup

Before you can really do anything with the build-package script, you have to define a configuration file. Project configuration files use the INI [1] file format. Every project configuration file must have a [build] section. The project configuration file for the MyServer would look something like this:

# MyServer.cfg

[build]
name = MyServer  # this has nothing to do with the package namespace
version = +
template = release.cfg
tag-layout = subfolder
upload-type = setup.py
package-index = https://pypi.projekt01.ch/private
package-index-username = username
package-index-password = password
buildout-server = https://pypi.projekt01.ch/++projects++/
buildout-server-username = username
buildout-server-password = password
svn-repos = https://svn.projekt01.ch/svn/myserver/packages/
svn-repos-username = somesvnuser
svn-repos-password = somepass
packages = p01.cdn
           p01.core
           p01.web

Let’s go over each of the settings in the build section of MyServer.cfg.

• name - This is the name of the project. It can be anything you want and has nothing to do with the packages that make up the project. The name will be part of the generation buildout configuration files.

• version - This is the version to use when making a new release of the Project. The version number becomes part of the filename for the generated buildout configuration files.

  • Using + as the version will simply increment the version number of the project from the versions that have already been released.

• template - This is a base buildout configuration file to use for all deployments. When a new Project release is created, the [versions] section will automatically be updated with the correct versions of each of the p01.* packages. More on this later.

• tag-layout - Choose from flat or subfolder

  • flat Tags will be created in svn as /tags/package-version This is the default setting.

  • subfolder Tags will be created in svn as /tags/package/version

• upload-type - Choose from internal or setup.py

  • internal Upload packages to a WebDAV enabled web server using the below credetials. This is the default setting. (actually does a python setup.py sdist and uploads the result)

  • setup.py Executes python setup.py sdist register upload, does nothing else as this command should take care of the upload.

• upload-format - setup.py release format option use as –formats argument. Choose from zip, gztar, bztar, ztar or tar. Note, there is only one format option allowed.

• package-index - The url to a WebDAV [2] enabled web server where generated eggs for each of the p01.* packages should be uploaded. Used for upload only if upload-type is internal. Also used to check/get existing versions of packages.

• package-index-username - The username for accessing the WebDAV server

• package-index-password - The password for accessing the WebDAV server

• buildout-upload-type - Choose from webdav, local or mypypi

  • webdav Upload generated buildout files to the url specified by buildout-server with the WebDAV protocol.

  • local Just generate buildout files, don’t upload them. If buildout-server is given buildout files will be copied to that folder.

  • mypypi Upload generated buildout files to the url specified by buildout-server. The url should point to the mypypi upload page. (Something like http://yourhost/++projects++/)

• buildout-server - The url to a WebDAV enabled web server where generated buildout files should be uploaded. If buildout-upload-type is local this is a path on the local filesystem. Buildout files wil be copied to this folder. If not given, the process stops after releasing the packages.

• buildout-server-username - The username for accessing the WebDAV server

• buildout-server-password - The password for accessing the WebDAV server

• svn-repos - The url for the subversion repository where all the source code lives, including release tags.

• svn-repos-username - The username for the url repository. Use the command line option --force-svnauth to force all svn operations to use this credential. Otherwise cached authentication will be used.

• svn-repos-password - The password for the url repository.

• hash-config-files - Add hashes based on file content to dependent config filenames.

• packages - a list of packages that are part of the project. These are the packages that live in the svn repository and that should be released in conjunction with each other.

Defining a Release Template

As we saw in the previous section, MyServer.cfg refers to a file called release.cfg. This is just a base buildout configuration. Additional to this, we can also define different configuration data as define in stage and production section. Such sections can get used in a product deployment as additional (shared) variables. For the MyServer project, it might look like this:

# release.cfg

[buildout]
extends = http://download.zope.org/zope3.4/3.4.0/versions.cfg
parts = test
find-links = https://pypi.projekt01.ch/private

[test]
recipe = zc.recipe.testrunner
eggs = p01.cdn
       p01.core
       p01.web

[app]
recipe = zc.zope3recipes:app
servers = zserver
site.zcml = <include package="p01.web" file="app.zcml" />
eggs = p01.web

[zope3]
location =

[stage]
memcached = 127.0.0.1:11211

[production]
memcached = 10.0.0.1:11211

When a new release of the MyServer project is made, a [versions] section will be added to this configuration file with all the correct p01.* versions pinned down.

Defining Multiple Deployment Configurations

Each time you release a Project, you may want to generate different buildout configuration files for all the different deployment environments you might have. For example, you may have three different environments: Development, Stage, and Production. These are called variants. Each environment may need to have the application run on different ports, at different log levels, or have other small differences.

We can easily generate additional configuration variants by adding addtional sections to the MyServer.cfg file:

# MyServer.cfg

[Development]
template = instance.cfg
vars = stage
port = 9080
logdir = /opt/myserver/dev/logs
install-dir = /opt/myserver/dev
loglevel = debug
cache-size = 1000

[Stage]
template = instance.cfg
vars = stage
port = 9082
logdir = /opt/myserver/stage/logs
install-dir = /opt/myserver/stage
loglevel = info
cache-size = 1000

[Production]
template = instance.cfg
vars = production
port = 8080
logdir = /var/log/myserver
install-dir = /opt/myserver/
loglevel = warn
cache-size = 200000

We can then have a single instance.cfg file that uses python’s built in string templating to access the variables we set in MyServer.cfg. For the MyServer project, it might look like this:

# instance.cfg

[buildout]
parts += server
directory = %(install-dir)s

[database]
recipe = zc.recipe.filestorage

[server]
recipe = zc.zope3recipes:instance
application = server-app
zope.conf =

  <product-config memcached>
    memcached %(memcached)s
  </product-config>

  <zodb>
    cache-size %(cache-size)s
    <filestorage>
      path ${database:path}
    </filestorage>
  </zodb>

  <server>
    type WSGI-HTTP
    address %(port)s
  </server>

  <eventlog>
    level %(loglevel)s
    <logfile>
      formatter zope.exceptions.log.Formatter
      path %(logdir)s/server.log
    </logfile>
  </eventlog>

  <accesslog>
    <logfile>
      level info
      path %(logdir)s/server-access.log
    </logfile>
  </accesslog>

As you can see, the MyServer.cfg uses additional vars (stage, production) which makes it very simple to define a lagrge amount of shared attributes in a release template and use them in the instance template. Note, a side effect from the python config parser is, that you will inherit arguments define in a duplicated section define in an template loaded via (buildout) extends.

Releasing a Project

Once you have created all the necessary configuration files, you can make your first project release. This is where the build-package script comes in to play. The first time you run the build-package script, the only option you need to pass in will be the configuration file.

The build-package script will prompt you for version information about each of the packages it will be releasing as part of the MyServer.cfg project. Your first interaction with the script might look like this:

$ build-package -c MyServer.cfg --quiet
Version for `p01.cdn` : 1.0.0
The release p01.cdn-1.0.0 does not exist.
Do you want to create it? yes/no [yes]: yes
Version for `p01.core` : 1.0.0
The release p01.core-1.0.0 does not exist.
Do you want to create it? yes/no [yes]: yes
Version for `p01.web` : 1.0.0
The release p01.web-1.0.0 does not exist.
Do you want to create it? yes/no [yes]: yes

The next time you make a release, you can set the -n flag for build-package to automatically guess the next version that should be released. It does this by first looking for all the release tags of a given package and finding the last changed revision for the trunk of a given package. If any code for the given package was changed since the last time it was released, it will bump the most minor version number automatically. If no change has occured, it will choose the latest existing release.

You can also use the -d flag to make build-package not prompt you before creating a new release.

If you need to make a new release from a particular branch, you can use the -b option. For example, if bug fixes have been made to the MyServer-1.x branch, we can create a new release using code from this branch like this:

$ build-package -c MyServer.cfg -nb MyServer-1.x

When the new package versions are calculated, they will be versioned along the 1.x line, even if you have since created 2.x releases, by analyzing the name of the branch.

Caveat when using -n and -d on a branch with a branchname ending with the version number is that you’ll need to have the package versions matching the branch version. E.g. having a branch: branches/myserver-1.9 will suppose packages like p01.core-1.9.x and p01.web-1.9.x and so on. You should be aware of this also when releasing packages from the trunk. Most probably you’ll drive development on the trunk and branch out for a stable. In this case package versions on the branch should be kept inline.

Install a Released Project

p01.build also comes with a very simple installation script that can be used to quickly install any variant of the released project:

$ deploy --help
Usage: deploy [options]

Options:
  -h, --help            show this help message and exit
  -u URL, --url=URL     The base URL at which the releases can be found.
  -p PROJECT, --project=PROJECT
                        The name of the project to be installed.
  -V VARIANT, --variant=VARIANT
                        The variant of the project to be installed.
  -v VERSION, --version=VERSION
                        The version of the project to be installed.
  --directory=FOLDER    Override installation target folder
  -l, --latest          When specified, the latest version will be chosen.
  --username=USER       The username needed to access the site.
  --password=PASSWORD   The password needed to access the site.
  -b PATH, --buildout-path=PATH
                        The path to the buildout executable.
  --quiet               When specified, no messages are displayed.
  --verbose             When specified, debug information is created.
  --timeout=TIMEOUT     Socket timeout passed on to buildout.

For example, to install the latest Stage version of the MyServer project, you would run:

$ deploy -u https://pypi.projekt01.ch/++projects++/ -p MyServer -V Stage –latest

Creating Helper Scripts

Sometimes it can be a pain to remember what all the command line options are that you need to pass for building your project. Fortunately, it is really easy to create helper scripts that just set some defaults for you.

For example, to create a build-myserver script, you would add the following to a buildout configuration file:

[uploads]
recipe = p01.recipe.setup:mkdir
path = ${buildout:directory}/parts/uploads

[build-myserver]
recipe = zc.recipe.egg
eggs = p01.build
scripts = build=build-myserver
initialization =
    sys.argv[1:1] = ['-c', 'MyServer.cfg',
                     '-o', '${buildout:directory}/parts/uploads']

The possibilities are endless!

0.6.0 (unreleased)

• feature: added uploadFormat build option for build zip or tar.gz release files Use the new uploadFormat option additional to the uploadType option in your project setup file. Use one of zip, gztar, bztar, ztar or tar. Note, there is only one format option allowed.

0.5.1 (2015-08-26)

• feature: added option -s, –storage-path which defines a directory where we store the created files before upload to pypi, webdav etc. This prevents that all files get generated into the package root.

• cleaup code, get rid of pkg_resources.parse_version deprecation message. Ported the parse_version method for now.

• removed p01/build/install.py and move code to p01/build/deploy.py. Switch the entry point for deploy to deploy.py

0.5.0 (2015-04-15)

• bugfix: parse given buildout parts content as is and only rstrip continue lines. This allows to use any content in parts like script content etc. without any intend troubles. The parse concept is similar and compatible with the one used in zc.buildout > 2.0

• This package provides a product build, release and installation system based on package versions using zc.buildout. The core concept is taken from keas.build and provides the same configuration syntax. The main difference is that we don’t setup the install script as install because there is a conflict with the gnu install script call on ubuntu (pycairo, python waf install) used by a popen recipe (p01.recipe.setup:popen). This implementation will offer the install script as a deploy entry_point. You can simply install the deploy script with easy_install p01.build on your server. For more information see p01/build/README.txt.

• initial branch of keas.build 0.4.1