mls.fabfile 0.4

Requirements

mls.fabfile currently uses knife to communicate with Rackspace servers. Please make sure knife is installed and configured successfully on your system.

Install

You can install mls.fabfile with PIP:

pip install mls.fabfile

All required dependencies will be installed automatically.

Usage

First, we need a working knife.rb file to interact with our Chef server and the Rackspace cloud eco system. Below is an example knife.rb file that gets all its required info from environment variables. This way you can add this knife.rb file inside a .chef directory to your project and savely put it under version control:

# Logging.
log_level                         :info
log_location                      STDOUT

# Chef server configuration.
chef_server_url                   "#{ENV['KNIFE_CHEF_SERVER']}"
client_key                        "#{ENV['KNIFE_CLIENT_KEY']}"
node_name                         "#{ENV['KNIFE_NODE_NAME']}"
validation_client_name            "#{ENV['KNIFE_VALIDATION_CLIENT_NAME']}"
validation_key                    "#{ENV['KNIFE_VALIDATION_CLIENT_KEY']}"
encrypted_data_bag_secret         "#{ENV['ENCRYPTED_DATA_BAG_SECRET_FILE']}"

# Rackspace API configuration.
knife[:rackspace_api_key]       = "#{ENV['RACKSPACE_API_KEY']}"
knife[:rackspace_api_username]  = "#{ENV['RACKSPACE_USERNAME']}"
knife[:rackspace_endpoint]      = "#{ENV['RACKSPACE_ENDPOINT']}"
knife[:rackspace_version]       = "#{ENV['RACKSPACE_VERSION']}"

Next, we need a fabfile.py. All we need to do is to import mls.fabfile to make the fabric commands available and the available environments we can work with from propertyshelf.fabfile.common.:

# -*- coding: utf-8 -*-
"""Sample MLS deployment script."""

from fabric import api

from mls.fabfile import *
from propertyshelf.fabfile.common.environments import *


# Definition of role names to be used.
api.env.role_database = 'mls_db'
api.env.role_frontend = 'mls_fe'
api.env.role_staging = 'mls_staging'
api.env.role_worker = 'mls_app'

# Definition of used Rackspace flavors (server sized) for our servers.
api.env.flavor_database = '5'
api.env.flavor_frontend = '2'
api.env.flavor_staging = '2'
api.env.flavor_worker = '3'

# Definition of node names to be used.
api.env.nodename_database = 'mls-db'
api.env.nodename_frontend = 'mls-fe'
api.env.nodename_staging = 'mls-staging'
api.env.nodename_worker = 'mls-app'

# The Rackspace server image we use. This is a Debian 6.0.6.
api.env.os_image = '92a28e50-181d-4fc7-a071-567d26fc95f6'

# MLS specific configuration.
api.env.domain = 'mls-example.com'
api.env.mls_customizations = ['mlsext.realtorcom', ]
api.env.mls_policy_enabled = True
api.env.mls_policy_package = 'mlspolicy.example'
api.env.mls_policy_package_url = 'git https://github.com/yourname/mlspolicy.example'

You can now use fabric to manage your MLS:

$ fab -l
Sample MLS deployment script.

Available commands:

    development                 Work locally with vagrant.
    production                  Work with the production environment.
    staging                     Work with the staging environment.
    bootstrap.bundle_db_fe_app  Bootstrap a new MLS bundle: Database, Frontend, App Worker.
    bootstrap.database          Bootstrap a new standalone database server.
    bootstrap.frontend          Bootstrap a new standalone frontend server.
    bootstrap.staging           Bootstrap a staging system.
    bootstrap.worker            Bootstrap a new standalone application worker.
    client.remove               Remove an existing MLS application client.
    client.restart              Restart the application client component.
    client.update               Update the client packages.
    database.backup             Perform a backup of Zope's data on the server.
    database.download_blobs     Download blob part of Zope's data from the server.
    database.download_data      Download the database files from the server.
    database.download_zodb      Download ZODB part of Zope's data from the server.
    database.restart            Restart the database component.
    database.restore            Restore an existing backup of Zope's data on the server.
    database.upload_blob        Upload blob part of Zope's data to the server.
    database.upload_data        Upload the database files to the server.
    database.upload_zodb        Upload ZODB part of Zope's data to the server.
    frontend.restart            Restart the frontend components.
    frontend.restart_haproxy    Restart the HA-Proxy load balancer component.
    frontend.restart_nginx      Restart the NginX web server component.
    frontend.restart_varnish    Restart the Varnish caching proxy component.
    roles.check                 Check if the required roles are available.
    roles.create_missing        Create missing roles on the chef server.

Before we can start it is a good idea to check if all roles we defined are available on the chef server:

$ fab roles.check
Role mls_fe NOT available.
Role mls_db NOT available.
Role mls_staging NOT available.
Role mls_app NOT available.

Done.

To create the missing roles based on our configuration, we simply have to do:

$ fab roles.create_missing
Created role mls_db
Created role mls_fe
Created role mls_app
Created role mls_staging

Done.

Now we can create our staging system:

$ fab bootstrap.staging
[localhost] local: knife rackspace server create -S mls-staging -N mls-staging -f 5 -I 92a28e50-181d-4fc7-a071-567d26fc95f6 -r role[rackspace],role[mls_staging] -E staging

...

Done.

Note that there can only be one staging system. If you try to add another one with the same name, you’ll get an error message:

$ fab bootstrap.staging

Fatal error: Server "mls-staging" already exists in environment "staging".

Aborting.

If you need a second one, you can adjust the node name manually:

$ fab bootstrap.staging:nodename=mls-staging2
[localhost] local: knife rackspace server create -S mls-staging2 -N mls-staging2 -f 5 -I 92a28e50-181d-4fc7-a071-567d26fc95f6 -r role[rackspace],role[mls_ni_staging] -E staging

...

Done.

You can now manage the single components:

$ fab staging frontend.restart
[x.x.x.x] Executing task 'frontend.restart'
[x.x.x.x] sudo: /etc/init.d/haproxy restart
[x.x.x.x] out: sudo password:

[x.x.x.x] out: Restarting haproxy: haproxy.
[x.x.x.x] out:

[x.x.x.x] sudo: /etc/init.d/varnish restart
[x.x.x.x] out: sudo password:
[x.x.x.x] out: Stopping HTTP accelerator: varnishd.
[x.x.x.x] out: Starting HTTP accelerator: varnishd.
[x.x.x.x] out:

[x.x.x.x] sudo: /etc/init.d/nginx restart
[x.x.x.x] out: sudo password:
[x.x.x.x] out: Restarting nginx: nginx.
[x.x.x.x] out:


Done.
Disconnecting from x.x.x.x... done.

We also support download of the database files for local testing:

$ fab production database.download_data
[x.x.x.x] Executing task 'database.download_data'
This will overwrite your local Data.fs. Are you sure you want to continue? [Y/n]
[localhost] local: mkdir -p var/filestorage
[localhost] local: mv var/filestorage/Data.fs var/filestorage/Data.fs.bak
[x.x.x.x] out: sudo password:
[x.x.x.x] sudo: rsync -a var/filestorage/Data.fs /tmp/Data.fs
[x.x.x.x] out: sudo password:
[x.x.x.x] out:
[x.x.x.x] download: /Volumes/Work/Propertyshelf/MLS/Provisioning/var/filestorage/Data.fs <- /tmp/Data.fs
This will overwrite your local blob files. Are you sure you want to continue? [Y/n]
[localhost] local: rm -rf var/blobstorage_bak
[localhost] local: mv var/blobstorage var/blobstorage_bak
[x.x.x.x] sudo: rsync -a ./var/blobstorage /tmp/
[x.x.x.x] out: sudo password:
[x.x.x.x] out:
[x.x.x.x] sudo: tar czf blobstorage.tgz blobstorage
[x.x.x.x] out: sudo password:
[x.x.x.x] out:
[x.x.x.x] download: /Volumes/Work/Propertyshelf/MLS/Provisioning/var/blobstorage.tgz <- /tmp/blobstorage.tgz

Warning: Local file /Volumes/Work/Propertyshelf/MLS/Provisioning/var/blobstorage.tgz already exists and is being overwritten.

[localhost] local: tar xzf blobstorage.tgz

Done.
Disconnecting from x.x.x.x... done.

Once we have local data files, we can upload them to our development environment (a vagrant VM):

$ fab development database.upload_data client.restart
[localhost] local: vagrant ssh-config | grep IdentityFile
[127.0.0.1:2222] Executing task 'database.upload_data'
This will overwrite your remote Data.fs. Are you sure you want to continue? [y/N] y
[127.0.0.1:2222] sudo: mkdir -p /tmp/upload
[127.0.0.1:2222] put: var/filestorage/Data.fs -> /tmp/upload/Data.fs
[127.0.0.1:2222] sudo: chown mls /tmp/upload/Data.fs
[127.0.0.1:2222] sudo: supervisorctl stop zeoserver
[127.0.0.1:2222] out: zeoserver: stopped
[127.0.0.1:2222] out:

[127.0.0.1:2222] sudo: mv var/filestorage/Data.fs var/filestorage/Data.fs.bak
[127.0.0.1:2222] sudo: mv /tmp/upload/Data.fs var/filestorage/Data.fs
This will overwrite your remote blob files. Are you sure you want to continue? [y/N] y
[127.0.0.1:2222] sudo: mkdir -p /tmp/upload
[localhost] local: tar czf blobstorage_upload.tgz blobstorage
[127.0.0.1:2222] put: var/blobstorage_upload.tgz -> /tmp/upload/blobstorage.tgz
[127.0.0.1:2222] sudo: chown mls /tmp/upload/blobstorage.tgz
[127.0.0.1:2222] sudo: tar xzf blobstorage.tgz
[127.0.0.1:2222] sudo: supervisorctl stop zeoserver
[127.0.0.1:2222] out: zeoserver: ERROR (not running)
[127.0.0.1:2222] out:

[127.0.0.1:2222] sudo: mv var/blobstorage var/blobstorage_bak
[127.0.0.1:2222] sudo: mv /tmp/upload/blobstorage var
[127.0.0.1:2222] sudo: supervisorctl start zeoserver
[127.0.0.1:2222] out: zeoserver: started
[127.0.0.1:2222] out:

[127.0.0.1:2222] Executing task 'client.restart'
[127.0.0.1:2222] sudo: supervisorctl restart application
[127.0.0.1:2222] out: application: stopped
[127.0.0.1:2222] out: application: started
[127.0.0.1:2222] out:


Done.
Disconnecting from 127.0.0.1:2222... done.

We can also get a list of nodes for already defined roles:

$ fab roles.list_nodes
Role: mls_fe
- mls-fe: x.x.x.x

Role: mls_db
- mls-db: x.x.x.x

Role: mls_staging
- mls-staging: x.x.x.x
- vagrant-mls-staging: 10.0.2.15

Role: mls_app
- mls-app-01: x.x.x.x


Done.

This can be useful if we want to execute a task only for a given node:

$ fab frontend.restart_nginx:hosts=x.x.x.x
[x.x.x.x] Executing task 'frontend.restart_nginx'
[x.x.x.x] sudo: /etc/init.d/nginx restart
[x.x.x.x] out: sudo password:
[x.x.x.x] out: Restarting nginx: nginx.
[x.x.x.x] out:


Done.
Disconnecting from x.x.x.x... done.

0.4 (2014-04-10)

• Added snapshotbackup option for database roles.

• Ability to update the maintenance client as well.

0.3 (2013-12-21)

• Moved common functionality to propertyshelf.fabfile.common.

• New tasks: rebuild client, list nodes for roles.

0.2 (2013-11-19)

• Set custom hostname for development environment so chef search works.

• Enabled data upload (Data.fs and blobs).

0.1.1 (2013-11-19)

• Added missing files.

0.1 (2013-11-19)

• Work with development, staging and production environments.

• Check for chef roles and create missing ones from configurations.

• Bootstrap MLS systems.

• Remove MLS clients if not needed any more.

• Manage database, frontend and worker components.

• Update client buildout packages.

• Download database files for local testing.