collective.hostout 1.0a5

Development buildout

For example, let’s say we had the worlds simplest wsgi application

from webob import Request, Response

def MainFactory(global_config, **local_conf):
    return MainApplication()

class MainApplication(object):
    """An endpoint"""

    def __call__(self, environ, start_response):
        request = Request(environ)
        response = Response("Powered by collective.hostout!")
        return response(environ, start_response)

We keep this is a package in src/hellowsgi. We will create a buildout cfg file called base.cfg

[buildout]
parts = demo pasterini
develop =
  src/hellowsgi

[demo]
recipe=zc.recipe.egg
eggs =
    PasteScript
    hellowsgi

[pasterini]
recipe = collective.recipe.template
output = parts/demo/paster.ini
port = 8080
input = inline:
    [server:main]
    use = egg:Paste#http
    host = 0.0.0.0
    port = ${:port}

    [pipeline:main]
    pipeline =
        app

    [app:app]
    use = egg:hellowsgi#main

Once we bootstrap and build this:

$> python bootstrap.py -c base.cfg
$> bin/buildout -c base.cfg

we have a working wsgi app if you run

$> bin/paster serve parts/demo/paster.ini

Production buildout

Next you will create a “production buildout” which extends your base.cfg. This might contain parts to install webservers, databases, caching servers etc.

Our prod.cfg is very simple

[buildout]
extends = base.cfg
parts += supervisor

[supervisor]
recipe=collective.recipe.supervisor
programs=
  10 demo ${buildout:directory}/bin/paster [serve ${pasterini:outout}] ${buildout:directory} true

[pasterini]
port = 80

Deployment buildout

Now create a 3rd buildout file, called buildout.cfg. This will be our development/deployment buildout

[buildout]
extends = base.cfg
parts += host1

[host1]
recipe = collective.hostout
host = myhost.com
hostos = ubuntu
user = myusername
path = /var/buildout/demo
buildout = prod.cfg
post-commands = bin/supervisord

This buildout part will install a script which will deploy prod.cfg along with hellowsgi to remote path /var/buildout/demo on our server myhost.com

$> bin/buildout
Develop: '.../example'
Uninstalling host1.
Installing demo.
Installing host1.

As part of the buildout process hostout will automatically determine the versions of all the eggs in your development buildout in a file called hostoutversions.cfg and will pin them for you during deployment. This ensures that the production buildout will be running the same software as you have tested locally. Remember to manually version pin any additional eggs you use in your prod.cfg as these will not be pinned for you.

Running hostout deploy for the first time

The bin/hostout command takes three kinds of parameters,

hostout [hostname(s)] [commands] [command arguments]

in our case we will run

$> bin/hostout host1 deploy

The first thing will do is ask you your password and attempt to login in to your server. It will then look for /var/buildout/demo/bin/buildout and when it doesn’t find it it will automatically run another hostout command called bootstrap.

Bootstrap is further broken down into three commands, bootstrap_users, bootstrap_python and bootstrap_buildout. These will create an additional buildout-user to build and run your application, install basic system packages needed to run buildout and install buildout into your remote path. It will attempt to detect which version of linux your server is running to os python, but if this fails it will attempt to compile python from source. The version of python used will match the major version of python which your development buildout uses.

Deploying and re-deploying

Once hostout bootstrap has ensured a working remote buildout, deployment will continue by running the following commands:

1. “uploadeggs”: Any develop eggs are released as eggs and uploaded to the server. These will be uploaded directly into the buildout’s buildout-cache/downloads/dist directory which buildout uses to find packages before looking up the package index. It’s very important the packages under development work when packaged, ie are capable of being packaged via “python setup.py sdist”. A common mistake is when relying on setuptools to automatically detect which files should be included but not have the correct setuptools SCM helpers installed if you are using git or hg e.g. for git do “easy_install setuptools-git”. This will also upload a “pinned.cfg” which contains the generated version numbers for the packages under development that have been uploaded.

2. “uploadbuildout”: The relevant .cfg files and any files/directories in the “include” parameter are synced to the remote server.

3. “buildout”: Upload a final “pinned.cfg” which includes the generated development package versions pins + all the versions of all the dependencies of the development buildout from where the system is being deployed from. These discovered pinned versions are recorded during the local buildout process by the hostout recipe and recorded in a local “hostoutversions.cfg” file. Buildout is then run on the remote production buildout.

If you continue to develop your application you can run hostout deploy each time and it will only upload the eggs that have changed and buildout will only reinstall changed parts of the buildout.

In our example above deployment would look something like this

$> bin/hostout host1 deploy
running clean
...
creating '...example-0.0.0dev_....egg' and adding '...' to it
...
Hostout: Running command 'predeploy' from 'collective.hostout'
...
Hostout: Running command 'uploadeggs' from 'collective.hostout'
Hostout: Preparing eggs for transport
Hostout: Develop egg src/demo changed. Releasing with hash ...
Hostout: Eggs to transport:
    demo = 0.0.0dev-...
...
Hostout: Running command 'uploadbuildout' from 'collective.hostout'
...
Hostout: Running command 'buildout' from 'collective/hostout'
...
Hostout: Running command 'postdeploy' from 'collective/hostout'
...

Now if you visit myhost.com you will see your web application shared with the world

Basic Options

host

the IP or hostname of the host to deploy to. by default it will connect to port 22 using ssh. You can override the port by using hostname:port

user

The user which hostout will attempt to login to your host as. Will read a users ssh config to get a default.

password

The password for the login user. If not given then hostout will ask each time.

identity-file

A public key for the login user.

extends

Specifies another part which contains defaults for this hostout

fabfiles

Path to fabric files that contain commands which can then be called from the hostout script. Commands can access hostout options via hostout.options from the fabric environment.

Deploy options

buildout

The configuration file you which to build on the remote host. Note this doesn’t have to be the same .cfg as the hostout section is in but the versions of the eggs will be determined from the buildout with the hostout section in. Defaults to buildout.cfg

path

The absolute path on the remote host where the buildout will be created. Defaults to ~${hostout:effective-user}/buildout

pre-commands

A series of shell commands executed as root before the buildout is run. You can use this to shut down your application. If these commands fail they will be ignored.

post-commands

A series of shell commands executed as root after the buildout is run. You can use this to startup your application. If these commands fail they will be ignored.

sudo-parts

Buildout parts which will be installed after the main buildout has been run. These will be run as root.

parts

Runs the buildout with a parts value equal to this

include

Additional configuration files or directories needed to run this buildout

buildout-cache

If you want to override the default location for the buildout-cache on the host

python-version

The version of python to install during bootstrapping. Defaults to version used in the local buildout.

hostos

Over which platform specific bootstrap_python command is called. For instance if hostos=redhat, bootstrap_python_redhat will be called to use “yum” to install python and other developer tools. This paramter is also used in hostout.cloud to pick which VM to create.

Users and logins

The bootstrap_users command is called as part of the bootstrap process which is called if no buildout has already been bootstraped on the remote server. This command will login using “user” (the user should have sudo rights) and create two additional users and a group which joins them.

effective-user

This user will own the buildouts var files. This allows the application to write to database files in the var directory but not be allowed to write to any other part of the buildout code.

buildout-user

The user which will own the buildout files. During bootstrap this user will be created and be given a ssh key such that hostout can login and run buildout using this account.

buildout-group

A group which will own the buildout files including the var files. This group is created if needed in the bootstrap_users command.

In addition the private key will be read from the location “identity_file” and be used to create a password-less login for the “buildout-user” account by copying the public key into the “authorized_keys” file of the buildout_user account. If no file exists for “identity_file” a DSA private key is created for you in the file “${hostname}_key” in the buildout directory. During a normal deployment all steps are run as the buildout-user so there is no need to use the “user” account and therefore supply a password. The exception to this is if you specify “pre-deploy”, “post-deploy” or “sudo-parts” steps or have to bootstrap the server. These require the use of the sudo-capable “user” account. If you’d like to share the ability to deploy your application with others, one way to do this is to simply checkin the private key file specified by “identity_file” along with your buildout. If you do share deployment, remember to pin your eggs in your buildout so the result is consistent no matter where it is deployed from. One trick you can use to achieve this is to add “hostoutversions.cfg” to the “extends” of your buildout and commit “hostoutversions.cfg” to your source control as well.

Making a hostout plugin

It can be very helpful to package up our fabfiles for others to use.

Hostout Plugins are eggs with three parts :-

1. Fabric script

2. A zc.buildout recipe to initialise the parameters of the fabric file commands

3. Entry points for both the recipe and the fabric scripts

>>>    entry_points = {'zc.buildout': ['default = hostout.myplugin:Recipe',],
...                    'fabric': ['fabfile = hostout.myplugin.fabfile']
...                    },

Once packaged and released others can add your plugin to their hostout e.g.

>>> write('buildout.cfg',
... """
... [buildout]
... parts = host1
...
... [host1]
... recipe = collective.hostout
... extends = hostout.myplugin
... param1 = blah
... """ )

>>> print system('bin/buildout')

>>> print system('bin/hostout host1')
cmdline is: bin/hostout host1 [host2...] [all] cmd1 [cmd2...] [arg1 arg2...]
Valid commands are:
...
   mycommand        : example of command from hostout.myplugin

Your fabfile can get access parameters passed in the commandline by defining them in your function. e.g.

def mycommand(cmdline_param1, cmdline_param2):
    pass

Your fabfile commands can override any of the standard hostout commands. For instance if you which your plugin to hook into the predeploy process then just add a predeploy function to your fabfile.py

def predeploy():
   api.env.superfun()

It is important when overridding to call the “superfun” function so any overridden functions are also called.

You can also call any other hostout functions from your command

def mycommand():
  api.env.hostout.deploy()

The options set in the buildout part are available via the Fabric api.env variable and also via “api.env.hostout.options”.

#TODO Example of echo plugin

Using fabric plugins

You use commands others have made via the extends option. Name a buildout recipe egg in the extends option and buildout will download and merge any fabfiles and other configuration options from that recipe into your current hostout configuration. The following are examples of built-in plugins. Others are available on pypi.

hostout.cloud

Will create VM instances automatically for you on many popular hosting services such as Amazon, Rackspace and Slicehost

hostout.supervisor

Will stop a supervisor before buildout is run and restart it afterwards. It provides some short commands to quickly manage your applications from your hostout commandline

1.0a5 (2011-03-21)

• Made deployment steps more discrete. uploadbuildout, uploadeggs and buildout make sense to be called by themselves. This changes the generated buildout .cfg files. [Dylan Jay]

• Commands can now return values and have named arguments [Dylan Jay]

• Changed how overridden commands called base command by introducing env.superfun [Dylan Jay, Adam Terrey]

• Allow args to the buildout command [Dylan Jay]

• added asbuildoutuser function decorator [Adam Terrey]

• added an apt-get update as part of the bootstrap [Adam Terrey]

• Put command and rsync command [Dylan Jay]

• fix bootstrap_python to use specific dot version. [Dylan Jay]

• Made file path names more robust when collecting files for hashing during getHostoutPackage [Adam Terrey]

• remove buildout as a dep so scripts won’t rebuild buildout script [Dylan Jay]

• Normal buildout should no longer need sudo user, just buildout-user. [Dylan Jay, Adam Terrey]

• Will bootstrap if no buildoutuser exists rather than ask for password [Adam Terrey]

• Fixed typos in docs [Jean Jordaan]

1.0a4 (2011-01-06)

• using decorators for setting user used for command

• cleaned up user adding code. renamed to bootstrap_users

• BUG: two hostout parts with different extends= values won’t create two separate command stacks but instead one combined one.

• BUG: extends was global to all hostouts [Fulvio Casali]

• merged hostout.ubuntu and hostout.redhat back into core. These are now deprecated.

• broke bootstrap step into bootstrap_users bootstrap_python bootstrap_buildout

• will install python from source if no specific os bootstrap_python available.

• change default buildout path to /var/buildout/PARTNAME

• hellowsgi demo included in code make for simple tests

• os specific bootstrap installs much less. only what’s needed for basic buildout

• move to the github collective.

• new “shell” parameter to specify which shell you want commands executed as.

• improvements to setowners including suid for directories.

• changes to make more freebsd friendly (Tom Lazar)

• setowners on every deployment

1.0a3 (unreleased)

• now using entrypoints for fabfiles

1.0a2 (2010-05-27)

• plugins can now call commands in other plugins

• use buildout-user and buildout-group to increase security of hosted code

• added setowners command to reset permissions

• added setaccess command to setup passwordless access

• moved hostout.ubuntu and hostout.supervisor to seperate plugins

• introduced initcommand to set user to particular plugin

• identify-file default to hostname_key so unique to host

• usign sdist zips instead of eggs to make cross platform

• python2.6 compatibility fixes

1.0a1 (2010-02-14)

• switched to python2.6 and fabric0.9

0.9.4 (2009-01-31)

• commandline changed to allow multsite and arguments

• mr.developer plugin

• supervisor plugin

• documentation clearer and more details

• ssh tests now working

• Nice listing of available commands

0.9.3

• Fixed how extends works

• fabric commands have hostout in environment rather than argument

• new command to reset permissions

• fab_hosts etc is set for all commands

• supervisor recipe now uses fabfile

0.9.2 (2009-09-26)

• can extend from recipe directly instead of a buildout part

• supervisor plugin recipe

• fixed version pinning for recipes

0.9.1 (2009-09-17)

• fix bug in lowering case of package names in versions

0.9 (2009-09-15)

• new commandline to allow deploying to multiple hosts at once

• ability to take defaults from another part using ‘extends’ option

• fabfiles option to create your own hostout commands

• moved hostout cfg generation to deployment time

• egg releases hash of contents as version numbers and won’t be built or uploaded unless changed

• new parts option to only install specified parts

• many option names simplified (backwards compatibility maintained)

• pre-commmands now runs before buildout initialisation

• buildout now run as ‘effective-user’ rather than root. pre and post commands still run as root

• eggs released directly to the download cache rather than a seperate dist directory

• version recognition only done once for all hostout parts

0.1.3 (2009-05-06)

• Fixed getting fabfile as a resource when packaged as an egg

• Allow for specifying extra configuration not in the buildout files

0.1.2 (2009-04-24)

Initial release. Basic uploading of eggs and running of remote buildout.