SlapOS recipes.
Project description
slapos.cookbook
Cookbook of SlapOS recipes.
testing
Unit tests for recipes can be found under slapos/test/recipe. To run the tests use provided unittest.defaultTestLoader inside slapos/test/test_recipe by invoking
python setup.py test –test-suite slapos.test.test_recipe.additional_tests
Changes
1.0.84 (2018-12-28)
Drop slapos.recipe:xvfb, use simple slapos.recipe:wrapper instead.
Drop slapos.recipe:seleniumrunner and slapos.recipe:firefox, they were not used.
Encode unicode to UTF-8 on slapos.recipe:request and slapos.recipe:slapconfiguration
1.0.75 (2018-09-04)
erp5_test: stop using erp5_test recipe
random: fix password generation with newlines
erp5testnode: enable password authentication for scalability test system
pbs: Ignore numerical IDs (UID/GID) when push
request: add requestoptional.serialised
1.0.65 (2018-06-22)
Automatic restart of services when configuration changes
erp5_test: define cloudooo-retry-count value in test
userinfo: expose values as string
1.0.62 (2018-04-10)
promise.plugin: new recipe for python promises plugin script generation
1.0.59 (2018-03-15)
librecipe.execute: fix convert process arguments to string formatting.
1.0.58 (2018-03-14)
generic.mysql: unregister UDFs before (re)adding UDFs
Remove obsolete/unused recipes.
neoppod: add support for new –dedup storage option.
Use inotify-simple instead of inotifyx.
erp5.test: remove duplicated code.
librecipe: bugfixes found by pylint, performance improvements, and major refactoring of executable wrappers.
GenericBaseRecipe.createWrapper: remove ‘comments’ parameter.
Drop the ‘parameters-extra’ option and always forward extra parameters.
wrapper: new ‘private-dev-shm’ option (useful for wendelin.core).
generic.cloudooo: OnlyOffice converter support odf.
erp5testnode: don’t tell git to ignore SSL errors.
1.0.53 (2017-09-13)
check_port_listening: workaround for shebang limitation, reduce to a single file
erp5.test: pass new –conversion_server_url option to runUnitTest
1.0.52 (2017-07-04)
wrapper: Add option to reserve CPU core
slapconfiguration: Recipe reads partitions resource file
neoppod: add support for new –disable-drop-partitions storage option
random: Fix the monkeypatch in random.py to incorporate the recent changes in buildout ‘get’ function
random: Add Integer recipe.
librecipe.execute: Notify on file moved
zero_knowledge: allow to set destination folder of configuration file
1.0.50 (2017-04-18)
pbs: Do not parallelize calculus when the heaviest task is IO
re6st-registry: Refactor integration with re6st registry
erp5testnode: make shellinabox reusing password file of pwgen
1.0.48 (2017-01-31)
random-recipe: add option create-once to prevent storage file deletion by buildout
1.0.45 (2017-01-09)
recipe: set default timeout of check url promise to 20 seconds
1.0.44 (2016-12-30)
pbs: handles the fact that some parameters are not present when slaves are down
recipe: allow usage of pidfile in wrapper recipe
sshd: fix generation of authorized_keys
1.0.43 (2016-11-24)
pbs: fixes trap command for dash intepreter
pbs: remove infinite loops from pbs scripts.
random.py: new file containing recipes generating random values.
testnode: disallow frontend access to all folders, avoiding publishing private repositories
1.0.41 (2016-10-26)
dcron: new parameter to get a random time, with a frequency of once a day
softwaretype: fix parse error on ‘+ =’ when using buildout 2
pbs: General Improvement and fixes.
1.0.35 (2016-09-19)
pbs: fix/accelerates deployment of resilient instances
recipe: new recipe to get a free network port
Remove url-list parameter to download fonts from fontconfig instance
1.0.31 (2016-05-30)
Implement cross recipe cache for registerComputerPartition
Fix workaround for long shebang (place script on bin)
1.0.30 (2016-05-23)
Implement a workarround for long shebang
Implement Validation for user inputs ssl certificates
1.0.25 (2016-04-15)
fixup slap configuration: provide instance and root instance title
1.0.22 (2016-04-01)
slap configuration: provide instance and root instance title
1.0.16 (2015-10.27)
kvm recipe: fix bugs dowload image and disk creation
1.0.14 (2015-10.26)
kvm recipe: Allow to set keyboard layout language used by qemu and VNC
simplehttpserver-recipe: fix encoding error
For older entries, see https://lab.nexedi.com/nexedi/slapos/blob/a662db75cc840df9d4664a9d048ef28ebfff4d50/CHANGES.rst
generic_cloudooo
The generic_cloudooo recipe helps you to deploy cloudooo services with their configuration files.
How to use generic_cloudooo?
Here is an example of a section to add in your software.cfg :
[cloudooo-configuration]
recipe = slapos.cookbook:generic_cloudooo
configuration-file = ${directory:etc}/cloudooo.cfg
wrapper = ${directory:services}/cloudooo
data-directory = ${directory:srv}/cloudooo
ip = 0.0.0.0
port = 1234
ooo-paster = ${directory:bin}/cloudooo_paster
mimetype_entry_addition =
text/html application/pdf wkhtmltopdf
openoffice-port = 1235
ooo-binary-path = ${directory:libreoffice-bin}/program
environment =
FONTCONFIG_FILE = ${fontconfig-instance:conf-path}
PATH = ${binary-link:target-directory}
ooo-uno-path = ${directory:libreoffice-bin}/basis-link/programWhere :
configuration-file is the path where the put the configuration file;
wrapper is the path where the put the final executable file;
data-directory is the folder where cloudooo would put it’s temporary files;
ip and port is where cloudooo will listen to;
ooo-paster is the path of the program that will load cloudooo configuration and start the application;
mimetype_entry_addition is additional entries to give to the default mimetype registry. (see section below.) The mimetype entry list is sorted in order to make the global mimetype at the bottom of the list. (i.e. * * ooo > text/* * ooo)
mimetype_entry_addition = <input_format> <output_format> <handler>openoffice-port is the port where the internal OpenOffice.org service will listen to;
ooo-binary-path is the path of the openoffice service executable file;
environment are environment vars to use with the openoffice binary;
ooo-uno-path is the path where UNO library is installed.
Default mimetype registry
application/vnd.oasis.opendocument* * ooo
application/vnd.sun.xml* * ooo
application/pdf text/* pdf
application/pdf * ooo
video/* * ffmpeg
audio/* * ffmpeg
application/x-shockwave-flash * ffmpeg
application/ogg * ffmpeg
application/ogv * ffmpeg
image/png image/jpeg imagemagick
image/png * ooo
image/* image/* imagemagick
text/* * ooo
application/zip * ooo
application/msword * ooo
application/vnd* * ooo
application/x-vnd* * ooo
application/postscript * ooo
application/wmf * ooo
application/csv * ooo
application/x-openoffice-gdimetafile * ooo
application/x-emf * ooo
application/emf * ooo
application/octet* * ooo
* application/vnd.oasis.opendocument* ooogeneric_varnish
This recipe creates a varnish instance dedicated for ERP5 with a web checker[1] set up.
How to Use generic_varnish ?
On slap console, you can instanciate varnish like this:
instance = request(
software_type='varnish',
partition_parameter_kw={
'backend-url':'https://[your_backend_address]:your_backend_port',
'web-checker-frontend-url':'http://www.example.com',
'web-checker-mail-address':'web-checker-result@example.com',
'web-checker-smtp-host':'mail.example.com',
}
)
backend-url is the backend url that varnish will cache.
web-checker-frontend-url is the entry-point-url that web checker will check the HTTP headers of all the pages in the web site.
web-checker-mail-address is the email address where web checker will send the HTTP Cache cheking result.
web-checker-smtp-host is the smtp server to be used to send the web checker result.
[Note] When web-checker-* parameters are not given, web_checker will be disabled.
References
[1] web_checker (it is a part of erp5.util) http://pypi.python.org/pypi/erp5.util web_checker: Web site HTTP Cache header checking tool
kvm
kvm: Introduction
The erp5.recipe.kvm aims to integrate KVM setups and buildout. This recipe is able to download one remote image and setup a KVM environment to use it.
This recipe is also capable to reuse images or partitions already present on disk to create the setup.
Examples
The follow examples lists different kind of configurations.
KVM with Remote and gzipped Image
[kvm-testing-with-remote-gzip-image] image = http://URL/public.mdv2010.0_x86_64.qcow2.img.gz # md5 checks are optional md5_download = adcff8adcff8adcff8adcff8 md5_image = 1a4e371a4e371a4e371a4e371a4e37 gzip = true # Use -hda instead -drive arg # Default is drive (see Options below) image_type = hda ### Common Configuration below. ### # VNC is optional kvm_vnc = <SOME-IP>:<VNC-DISPLAY> # Graphic is optional kvm_graphic = std # Define list of redirections. kvm_redir = tcp:13480::80 kvm_net = user,hostname=publicmandriva nic,model=ne2k_pci # This automatically create a redirection for 13456 -> 22 ssh_port = 13456 ssh_key_path = /path/to/mykey.key
KVM with Remote and raw Image
[kvm-testing-with-remote-raw-image] image = http://URL/public.mdv2010.0_x86_64.qcow2.img md5_download = 1a4e371a4e371a4e371a4e371a4e37 md5_image = 1a4e371a4e371a4e371a4e371a4e37 gzip = false ### The Rest Same configuration as previous ###
KVM with direct local Image file
This does not copy and/or download the image:
[kvm-testing-with-local-image] file = /home/you/public.mdv2010.0_x86_64.qcow2.img md5_image = 1a4e371a4e371a4e371a4e371a4e37 ### The Rest Same configuration as previous ###
KVM with a linux partition
This does not copy and/or download the image:
[kvm-testing-a-linux-partition] file = /dev/sdb ### The Rest Same configuration as previous ###
Options
location
When define, it does not use buildout parts directory to allocate the image.
image
URL to a remote image.
file
Use file makes recipe ignore image option. You can define a path to a image or partion to use.
image_type
You can define how the KVM will use the image with “-hdx” or “-drive”. By default it uses drive and the result is:
“kvm -drive file=IMAGE,if=virtio,cache=none,boot=on …”
if you use image_type=hda:
“kvm -hda IMAGE …”
gzip
If true means the image is compressed (gzipped), and the recipe will uncompress the image before copy it.
md5_download
When defined, this values is used to verify the file downloaded.
md5_image
When defined, this values is used to verify the image generated, don’t use it when a partition is used in file parameter.
kvm_vnc
Define the ip-address:display used by KVM to start the VNC server. If not defined, no VNC port is created.
kvm_redir
Define port redirections to the buildout.
kvm_graphic
If defined it adds the “-vga value” at the KVM command.
kvm_net
Define the net definitions, each value defines one “-net” in kvm command. Example:
- kvm_net =
user,hostname=publicmandriva nic,model=ne2k_pci
It generates:
“kvm -net user,hostname=publicmandriva -net nic,model=ne2k_pci …”
kvm_snapshot
Use “-snapshot” when run a KVM. This not write the changes direct into the image. Default value is False.
ssh_port
If defined creates a new redirection for port 22 and creates few script to connect to the instance.
ssh_hostname
By default it uses localhost. You don’t need to define this.
ssh_key_path
Path to the ssh key used to connect without password to the image running.
ssh_user
Define the server that will be used to connect to the instance.
kvm_bin_directory
Place where the scripts will be created. By default it uses bin-directory from buildout.
kvm_run_directory
Place where the pid file will be created, by default it uses var-directory from buildout.
Generated Commands
Few scripts are generated to you manage your KVM instance. The scripts names are created with the followed standard:
KVM-PARTS-NAME-ctl
Commands usage
KVM-PARTS-NAME-ctl (start|stop|status|restart)
This script is used to manage the KVM instance.
KVM-PARTS-NAME-sendfile REMOTEFILE LOCALFILE
Copy the local file to a remote place.
KVM-PARTS-NAME-getfile REMOTEFILE LOCALFILE
Copy the remote file to a local place.
KVM-PARTS-NAME-runscript COMMAND
Run a command into remote KVM computer.
kvm_frontend
kvm_frontend: Introduction
The slapos.recipe.kvm_frontend aims to provide proxy server to KVM instances.
It allows HTTPS IPv4/IPv6 proxying (with or without domain name), and supports the WebSocket technology needed for VNC-in-webapplication noVNC.
It works following the master/slave instances system. A master instance is created, containing all what is needed to run the proxy. Slave instances are later created, adding one line in the master instance’s proxy configuration that specify the IP/port to proxy to the KVM. The slave instance (kvm) is then accessible from http://[masterinstanceIPorhostname]/[randomgeneratednumber]
Instance parameters
Incoming master instance parameters
port - Port of server, optional, defaults to 4443.
domain - domain name to use, optional, default to host.vifib.net.
redirect_plain_http - if value is one of ['y', 'yes', '1', 'true'], will try to create a simple http server on port 80 redirecting to the proxy. Optional.
Incoming slave instance parameters
host - KVM instance IP or hostname. Mandatory.
port - KVM instance port, Mandatory.
https - if value is one of ['n', 'no', '0', 'false'], will try to connect to target in plain http. Optional.
Connection parameters
Outgoing master connection parameters
domain_ipv6_address - Proxy IP
site_url - Proxy URL
Outgoing slave connection parameters
site_url - URL of instance
domain_name - Domain name of proxy
port - Port of proxy
lamp
The lamp recipe helps you to deploy simply a php based application on slapos. This recipe is able to setup mariadb, apache and apache-php for your php application, and is also capable of configuring your software during installation to ensure a full compatibility.
How to use lamp ?
just add this part in your software.cfg to use the lamp.simple module
[instance-recipe]
egg = slapos.cookbook
module = lamp.simpleyou also need to extend lamp.cfg
extends =
https://lab.nexedi.com/nexedi/slapos/raw/slapos-0.50/stack/lamp.cfglamp.runner
When you install some software (such as prestashop) you need to remove or rename folder, with slapos you can not access to the www-data directory. to do this, you need to tell to lamp recipe to remove or/and it when software will be instantiated. Some software requires more than rename or delete a folder (manualy create database etc…) in this case you need to write a python script and lamp recipe must run it when installing your software.
How to use lamp.runner ?
this part of lamp recipe work with slapos.toolbox, Therefore you must add it to your recipe. in software.cfg, replace instance-recipe-egg part by
[instance-recipe-egg]
recipe = zc.recipe.egg
eggs =
${python-mysqlclient:egg}
${instance-recipe:egg}
slapos.toolbox[lampconfigure]and add into your instance.cfg
lampconfigure_directory = ${buildout:bin-directory}/lampconfigureCONDITION
the action (delete, rename, script, chmod) only starts when the condition is filled. in instance.cfg, add:
file_token = path_of_file
and the action will begin when path_of_www-data/path_of_file will be created you can also use database to check condition. add
table_name = name_of_table constraint = sql_where_condition
name_of_table is the full or partial name(in some cases we can not know the prefix used to create tables) of table into mariadb databse for example table_name = admin. if you use name_of_table = **, the action will begin when database is ready. constraint is the sql_condition to use when search entry into name_of_table for example constraint = admin_id=1
you can’t use file_token and table_name at the same time, otherwise file_token will be used in priority. Beware of conditions that will never be satisfied.
ACTION
The action starts when condition is true
1- delete file or folder into instance.cfg, use:
delete = file_or_folder1, file_or_folder2, file_or_folder3 ...
for example delete = admin
2- rename file or folder into instance.cfg, use
rename_chmod = mode (optional) rename = old_name1 => new_name1, old_name2 => new_name2, …
you can also use rename = old_name1, old_name2 => new_name2, … in this case old_name1 will be rename and the new name will be chose by joining old_name1 and mysql_user: this should give rename = old_name1 => old_name1-mysql_user, old_name2 => new_name2, … use rename_chmod to apply a mode to new_name1, new_name2, …
3- Change mode of file or directory
this is usefull if you want to change the mode of files into your www-data, use
chmod = file1, file2, …. mode = mode_to_apply (ex= 0644)
4- Launch python script
use script = ${configure-script:location}/${configure-script:filename} into instance.cfg, add part configure-script into software.cfg
parts = configure-script
[configure-script]
recipe = hexagonit.recipe.download
location = ${buildout:parts-directory}/${:_buildout_section_name_}
url = url_of_script_name.py
filename = script_name.py
download-only = TrueThe script_name.py should contain a main function, sys.argv is given to the main. you can write script_name.py like this:
...
def setup(args):
mysql_port, mysql_host, mysql_user, mysql_password, mysql_database, base_url, htdocs = args
.......
if __name__ == '__main__':
setup(sys.argv[1:])
base_url: is the url of php software
htdocs: is the path of www-data directory
mysql_user, mysql_password, mysql_database, mysql_host: is the mariadb parameters
you can also use “import MySQLdb” if you want to access to database via your python script
libcloud
Slapified recipe to interact with any libcloud supported IaaS system
librecipe
Thanks to using slapos.cookbook:librecipe it is easier to create zc.buildout recipes in SlapOS environment.
How to use?
In setup.py of recipe add only one install requires to slap.lib.recipe.
In code itself subclass from slap.lib.recipe.BaseSlapRecipe.BaseSlapRecipe.
Use _install hook:
from slap.lib.recipe.BaseSlapRecipe import BaseSlapRecipe
class Recipe(BaseSlapRecipe):
...
def _install(self):
# refer below for list of available objects
specific code
of recipe
Available variables self.:
name and options passed by zc.buildout during init
work_directory – buildout’s directory
bin_directory – places for generated binaries
running_wrapper_location – filename of wrapper to create
data_root_directory – directory container for data – inside this directory it is advised to create named directories for provided servers which needs data
backup_directory – directory container for backups – inside this directory it is advised o created named directories for backups, with same structure as in data_root_directory
var_directory – container for various, unix following things:
log_directory – container for logs
run_directory – container for pidfiles and sockets
etc_directory – place to put named files and directories of configuration for provided servers
computer_id – id of computer
computer_partition_id – if of computer partition
server_url - url of Vifib server
software_release_url – url of software release being instantiated
slap – initialised connection to Vifib server
computer_partition – initialised connection to computer partition
request – shortcut to computer partition request method
By default all directories are created before calling _install hook.
_install method shall return list of paths which are safe to be removed by buildout during part uninstallation.
Important assumptions
Because in SlapOS environment zc.buildout does not know when data are changed, recipes shall be always uninstalled/installed. This is done during constructing recipe instance which subclasses from BaseSlapRecipe.
mkdirectory
mkdirectory loops on its options and create the directory joined
The created directory won’t be added to path list.
These recipes provide the ability to save some buildout parameters and their value in a custom file, inside the instance folder.
In both recipes, you HAVE TO give a filename, which will be stored at the root of the instance folder
WriteRecipe :
Is used to create a section (named according to the buildout section_name).
You can give then as much parameters you wish, with their default values.
Whenever you run buildout, if the parameter has yet been saved in the config file, it will do nothing.
If the parameter’s value has changed in the config file, it won’t be overwritten
/!If you decide to change the default value of one parameter, ALL other parameters will be reseted in the config file, even if you changed it manually. Explanation : The default values aren’t expected to change, except while development purposes.
ReadRecipe :
It fills its own section with all the options in all the sections of the config file.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
