slapos.recipe.build 0.33

Pure download

Note: deprecated entry-point.

[buildout]
parts =
  download

[download]
recipe = slapos.recipe.build:download
url = https://some.url/file

Such profile will download https://some.url/file and put it in buildout:parts-directory/download/download

filename parameter can be used to change destination named filename.

destination parameter allows to put explicit destination.

md5sum parameter allows pass md5sum.

mode (octal, so for rw-r–r– use 0644) allows to set mode

Exposes target attribute which is path to downloaded file.

Simple clone

Only repository parameter is required. For each buildout run, the recipe will pick up the latest commit on the remote master branch:

>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = git-clone
...
... [git-clone]
... recipe = slapos.recipe.build:gitclone
... repository = http://git.erp5.org/repos/slapos.recipe.build.git
... use-cache = true
... """)

This will clone the git repository in parts/git-clone directory. Then let’s run the buildout:

>>> print system(buildout)
Installing git-clone.
Cloning into '/sample-buildout/parts/git-clone'...

Let’s take a look at the buildout parts directory now:

>>> ls(sample_buildout, 'parts')
d buildout
d git-clone

When updating, it will do a “git fetch; git reset @{upstream}”:

>>> print system(buildout)
Updating git-clone.
Fetching origin
HEAD is now at ...

Specific branch

You can specify a specific branch using branch option. For each run it will take the latest commit on this remote branch:

>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = git-clone
...
... [git-clone]
... recipe = slapos.recipe.build:gitclone
... repository = http://git.erp5.org/repos/slapos.recipe.build.git
... branch = build
... """)

Then let’s run the buildout:

>>> print system(buildout)
Uninstalling git-clone.
Running uninstall recipe.
Installing git-clone.
Cloning into '/sample-buildout/parts/git-clone'...

Let’s take a look at the buildout parts directory now:

>>> ls(sample_buildout, 'parts')
d buildout
d git-clone

And let’s see that current branch is “build”:

>>> import subprocess
>>> cd('parts', 'git-clone')
>>> print subprocess.check_output(['git', 'branch'])
* build

When updating, it will do a “git fetch; git reset build”:

>>> cd(sample_buildout)
>>> print system(buildout)
Updating git-clone.
Fetching origin
HEAD is now at ...

Specific revision

You can specify a specific commit hash or tag using revision option. This option has priority over the “branch” option:

>>> cd(sample_buildout)
>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = git-clone
...
... [git-clone]
... recipe = slapos.recipe.build:gitclone
... repository = http://git.erp5.org/repos/slapos.recipe.build.git
... revision = 2566127
... """)

Then let’s run the buildout:

>>> print system(buildout)
Uninstalling git-clone.
Running uninstall recipe.
Installing git-clone.
Cloning into '/sample-buildout/parts/git-clone'...

Let’s take a look at the buildout parts directory now:

>>> ls(sample_buildout, 'parts')
d buildout
d git-clone

And let’s see that current revision is “2566127”:

>>> import subprocess
>>> cd(sample_buildout, 'parts', 'git-clone')
>>> print subprocess.check_output(['git', 'rev-parse', '--short', 'HEAD'])
2566127

When updating, it will do a “git fetch; git reset revision”:

>>> cd(sample_buildout)
>>> print system(buildout)
Updating git-clone.
...
HEAD is now at 2566127 ...
...
<BLANKLINE>

Empty revision/branch

Specifying an empty revision or an empty branch will make buildout ignore those values as if it was not present at all (allowing to easily extend an existing section specifying a branch):

>>> cd(sample_buildout)
>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = git-clone
...
... [git-clone-with-branch]
... recipe = slapos.recipe.build:gitclone
... repository = http://git.erp5.org/repos/slapos.recipe.build.git
... revision = 2566127
...
... [git-clone]
... <= git-clone-with-branch
... revision =
... branch = master
... """)

>>> print system(buildout)
Uninstalling git-clone.
Running uninstall recipe.
Installing git-clone.
Cloning into '/sample-buildout/parts/git-clone'...

>>> cd(sample_buildout, 'parts', 'git-clone')
>>> print system('git branch')
* master

Revision/branch priority

If both revision and branch parameters are set, revision parameters is used and branch parameter is ignored:

>>> cd(sample_buildout)
>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = git-clone
...
... [git-clone]
... recipe = slapos.recipe.build:gitclone
... repository = http://git.erp5.org/repos/slapos.recipe.build.git
... branch = mybranch
... revision = 2566127
... """)

>>> print system(buildout)
Uninstalling git-clone.
Running uninstall recipe.
Installing git-clone.
Warning: "branch" parameter with value "mybranch" is ignored. Checking out to revision 2566127...
Cloning into '/sample-buildout/parts/git-clone'...
HEAD is now at 2566127 ...

>>> cd(sample_buildout, 'parts', 'git-clone')
>>> print system('git branch')
* master

Setup a “develop” repository

If you need to setup a repository that will be manually alterated over time for development purposes, you need to make sure buildout will NOT alter it and NOT erase your local modifications by specifying the “develop” flag:

[buildout]
parts = git-clone

[git-clone]
recipe = slapos.recipe.build:gitclone
repository = https://example.net/example.git/
develop = true

>>> cd(sample_buildout)
>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = git-clone
...
... [git-clone]
... recipe = slapos.recipe.build:gitclone
... repository = http://git.erp5.org/repos/slapos.recipe.build.git
... develop = true
... """)

>>> print system(buildout)
Uninstalling git-clone.
Running uninstall recipe.
Installing git-clone.
Cloning into '/sample-buildout/parts/git-clone'...

Buildout will then keep local modifications, instead of resetting the repository:

>>> cd(sample_buildout, 'parts', 'git-clone')
>>> print system('echo foo > setup.py')

>>> cd(sample_buildout)
>>> print system(buildout)
Updating git-clone.
Fetching origin
...
<BLANKLINE>


>>> cd(sample_buildout, 'parts', 'git-clone')
>>> print system('cat setup.py')
foo

Then, when update occurs, nothing is done:

>>> cd(sample_buildout, 'parts', 'git-clone')
>>> print system('echo kept > local_change')

>>> print system('git remote add broken http://git.erp5.org/repos/nowhere')
...

>>> cd(sample_buildout)
>>> print system(buildout)
Updating git-clone.
Fetching origin
Fetching broken
Unable to update:
Traceback (most recent call last):
...
CalledProcessError: Command '['git', 'fetch', '--all']' returned non-zero exit status 1
<BLANKLINE>
...
fatal: unable to access 'http://git.erp5.org/repos/nowhere/': The requested URL returned error: 500
error: Could not fetch broken
<BLANKLINE>

>>> cd(sample_buildout, 'parts', 'git-clone')
>>> print system('cat local_change')
kept

In case of uninstall, buildout will keep the repository directory:

>>> cd(sample_buildout)
>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = git-clone
...
... [git-clone]
... recipe = slapos.recipe.build:gitclone
... repository = http://git.erp5.org/repos/slapos.recipe.build.git
... develop = true
... # Triggers uninstall/install because of section signature change
... foo = bar
... """)

>>> print system(buildout)
Uninstalling git-clone.
Running uninstall recipe.
You have uncommited changes in /sample-buildout/parts/git-clone. This folder will be left as is.
Installing git-clone.
destination directory already exists.
...
<BLANKLINE>

Specific git binary

The default git command is git, if for a any reason you don’t have git in your path, you can specify git binary path with git-command option.

Ignore SSL certificate

By default, when remote server use SSL protocol git checks if the SSL certificate of the remote server is valid before executing commands. You can force git to ignore this check using ignore-ssl-certificate boolean option:

[buildout]
parts = git-clone

[git-clone]
recipe = slapos.recipe.build:gitclone
repository = https://example.net/example.git/
ignore-ssl-certificate = true

Other options

shared

Clone with --shared option if true. See git-clone command.

sparse-checkout

The value of the sparse-checkout option is written to the $GITDIR/info/sparse-checkout file, which is used to populate the working directory sparsely. See the SPARSE CHECKOUT section of git-read-tree command. This feature is disabled if the value is empty or unset.

Full example

[buildout]
parts = git-clone

[git-binary]
recipe = hexagonit.recipe.cmmi
url = http://git-core.googlecode.com/files/git-1.7.12.tar.gz

[git-clone]
recipe = slapos.recipe.build:gitclone
repository = http://example.net/example.git/
git-command = ${git-binary:location}/bin/git
revision = 0123456789abcdef

Basic example

Here is example to install one or several modules:

[buildout]
parts = perl-modules

[perl-modules]
recipe = slapos.recipe.build:cpan
modules =
  Class::Date
  Other::Module
# Optional argument specifying perl buildout part, if existing.
# If specified, recipe will use the perl installed by buildout.
# If not specified, will take the globally available perl executable.
perl = perl

Specific version

Note that cpan won’t allow you to specify version and will always take latest version available. To choose a specific version, you will need to specify the full path in cpan like in

[buildout]
parts = perl-modules

[perl-modules]
recipe = slapos.recipe.build:cpan
modules =
  D/DL/DLUX/Class-Date-1.1.10.tar.gz
perl = perl

Basic example

Here is example to install one or several modules:

[buildout]
parts = node-package

[node-package]
recipe = slapos.recipe.build:npm
modules =
  colors
  express

# Optional argument specifying perl buildout part, if existing.
# If specified, recipe will use the perl installed by buildout.
# If not specified, will take the globally available perl executable.
node = node-0.6

Specific version

[buildout]
parts = node-package

[node-package]
recipe = slapos.recipe.build:npm
modules =
  express@1.0.2
node = node-0.6

Example

[vm-run-environment]
PATH = ${openssh:location}/bin:${qemu:location}/bin:%(PATH)s

[vm-run-base]
recipe = slapos.recipe.build:vm.run
environment = vm-run-environment
mem = 256 * (${:smp} + 1)
smp = 4

Options

location

Produced files: <dist>.img (1 for each token of dists), passwd and optionally ssh.key

arch

QEMU architecture (the recipe runs the qemu-system-<arch> executable). It is also used to select the ISO in the sections refered by dists. Default to host architecture.

dists

List of VMs to build: each token refers to a buildout section name that describes the ISOs to use. See ISO sections below. Tokens can’t contain ‘.’ characters.

size

Size of the VM image. This must be an integer, optionally followed by a IEC or SI suffix.

mem

Default: 256

[<dist>/]preseed.<preseed>

Set the <preseed> value for the installation. The recipe has many default preseed values: you can see the list in the InstallDebianRecipe.preseed class attribute (file slapos/recipe/vm.py). Aliases are recognized (but the recipe includes a mapping that may be out-of-date.). Any value except passwd/* can optionally be prefixed so that they only apply for a particular VM.

[<dist>/]debconf.<owner>

List of debconf value for <owner> (usually a package name), each line with 2 whitespace-separated parts: <key> <value>. Like for preseed.* values, they can be specific to <dist>.

late-command

Shell commands to execute at the end of the installation. They are run inside the target system. This is a reliable alternative to the preseed.preseed/late_command option. The DIST shell variable is set to the VM being built.

packages

Extra packages to install. Like for late-command, do not use preseed.pkgsel/include. If you want to install packages only for some specific <dist>, you can do it in late-command, by testing $DIST and using apt-get install -y.

vm.run

Boolean value that is true by default, to configure the VM for use with the slapos.recipe.build:vm.run recipe:

• make sure that the ssh and sudo packages are installed

• an SSH key is automatically created with ssh-keygen, and it can be used to connect as root

ISO sections

<arch>.iso

Name of the section that provides the ISO image, for example by downloading it. This section must define 2 options: location is the folder containing the ISO, and filename is the file name of the ISO.

<arch>.kernel

Path to kernel image inside the ISO.

<arch>.initrd

Path to initrd image inside the ISO.

User setup

By default, there’s no normal user created. Another rule is that a random password is automatically generated if there is no password specified.

You have nothing to do if you only plan to use the VM with vm.run.

For more information about the passwd/* preseed values, you can look at the user-setup-udeb package at https://anonscm.debian.org/cgit/d-i/user-setup.git/tree/ and in particular the user-setup-ask and user-setup-apply scripts.

Example

[vm-install-environment]
# vm-run-environment refers to the section in common options
PATH = ${file:location}/bin:${p7zip:location}/bin:${vm-run-environment:PATH}

[vm-debian]
recipe = slapos.recipe.build:vm.install-debian
environment = vm-install-environment
dists = debian-jessie debian-stretch
size = 2Gi
late-command =
# rdnssd causes too much trouble with QEMU 2.7, because the latter acts as
# a DNS proxy on both IPv4 and IPv6 without translating queries to what the
# host supports.
  dpkg -P rdnssd
debconf.debconf =
  debconf/frontend noninteractive
  debconf/priority critical
# minimal size
preseed.apt-setup/enable-source-repositories = false
preseed.recommends = false
preseed.tasks =

[debian-jessie]
x86_64.iso = debian-amd64-netinst.iso
x86_64.kernel = install.amd/vmlinuz
x86_64.initrd = install.amd/initrd.gz

[debian-stretch]
<= debian-jessie
x86_64.iso = debian-amd64-testing-netinst.iso

[debian-amd64-netinst.iso]
...

Options

location

Folder where to store any produce file. Inside the guest, it is pointed to by the PARTDIR environment variable. It is also used as temporary storage for changes to the VM image.

vm

Folder containing the VM images and the ssh.key` file. See the location option of the vm.install-* recipes.

dist

VM image to use inside the vm folder.

commands

List of <command> options, each one being a shell script to execute via SSH. They are processed in sequence. This is usually only required if you want to reboot the VM. Default: command

mount.<name>

Extra mount point. The value is a host folder that is mounted as /mnt/<name>.

stop-ssh

Tell reboot function how to stop SSH (see Helpers). Default: systemctl stop ssh

user

Execute commands with this user. The value can be root. By default, it is empty and it means that:

• a slapos user is created with the same uid/gid than the user using this recipe on the host, which can help accessing mount points;

• sudo must be installed and the created user is allowed to become root without password.

In any case, SSH connects as root.

wait-ssh

Time to wait for (re)boot. The recipe fails if it can’t connect to the SSH server after this number of seconds. Default: 60

Helpers

Before commands are executed, all mount.<name> are mounted and a few helpers are set to make scripting easier.

set -e

This is done before anything else, to make buildout abort if any untested command fails.

reboot

Function to safely reboot the guest. The next command in commands will be executed once the SSH server is back.

map <host_path>

Function to map a folder inside ${buildout:directory}.

PARTDIR

Folder where to store any produced file. Inside the guest, it actually maps to location on the host. This is useful because you can’t write PARTDIR=`map ${:location}` if you don’t explicitly set location.

Example

[vm-run-base]
# extends above example in common options
vm = ${vm-debian:location}
dist = debian-jessie

[vm-debian]
# extends above example in vm.install-debian
packages += build-essential devscripts equivs git

[userhosts-repository]
recipe = slapos.recipe.build:gitclone
repository = https://lab.nexedi.com/nexedi/userhosts.git
# we don't need a working directory on the host
sparse-checkout = /.gitignore

[build-userhosts-map]
<= vm-run-base
repository = `map ${userhosts-repository:location}`
command =
  git clone -s ${:repository} userhosts
  cd userhosts
  mk-build-deps -irs sudo -t 'apt-get -y'
  dpkg-buildpackage -uc -b -jauto
  cd ..
  mv *.changes *.deb $PARTDIR

# Alternate way, which is required if [userhosts-repository] is extended
# in such way that the repository is outside ${buildout:directory}.
[build-userhosts-mount]
<= build-userhosts-map
mount.userhosts = ${userhosts-repository:location}
repository = /mnt/userhosts

[test-reboot]
<= vm-run-base
commands = hello world
hello =
  uptime -s
  echo Hello ...
  reboot
world =
  uptime -s
  echo ... world!