djc.recipe2 2.0

Complete example

Let’s now put into our settings file (myproject_site_config/settings.py) some more sane values:

>>> write('sites', 'myproject_site_config', 'settings.py', '''
... DATABASES = {
...     'default': {
...         'ENGINE': 'django.db.backends.sqlite3',
...         'NAME': 'storage.db'
...     }
... }
... TIME_ZONE = 'Europe/Rome'
... ''')

Now, in order for these settings to take effect, we don’t have to re-run buildout, as the import that the generated file does will pick them up:

>>> print system('bin/django diffsettings')
DATABASES = {'default': {'ENGINE': 'django.db.backends.sqlite3', 'NAME': 'storage.db'}}
SECRET_KEY = '...'
SETTINGS_MODULE = 'myproject_part_site.settings'  ###
TIME_ZONE = 'Europe/Rome'

Seems like it worked!

We decided to put the database in a SQLite file named storage.db, which currently doesn’t exist:

>>> isfile('storage.db')
False

Let’s now tell Django to create the database:

>>> print system('bin/django syncdb --noinput')
Creating tables ...
Installing custom SQL ...
Installing indexes ...
Installed 0 object(s) from 0 fixture(s)
<BLANKLINE>

And we will see that the database has been created:

>>> isfile('storage.db')
True

Debug mode

We can now start developing but, sooner or later, we’ll recognize that we haven’t set DEBUG = True, which is fundamental if your name is not Donald Knuth.

We could add it straight away in myproject_site_config/settings.py, but that might cause problems when we’re Going production, because you definitely want to have DEBUG and its sisters off when you’re out in the open.

Therefore, another option that we have is to do the following:

>>> write('buildout.cfg', '''
... [buildout]
... parts =
...     myproject
...
... [myproject]
... recipe = djc.recipe2
... settings-override =
...     DEBUG = True
...     TEMPLATE_DEBUG = True
... ''')

Anything that we put in settings-override will be appended at the end of the buildout-generated settings.py (treated as a string, so beware that no correctness checking is performed). This allows us to quickly differentiate production and development buildouts without having the need to come up with two different settings.py files (one for production and one for development).

If we re-run the buildout and look at the results, we will see that we are now in debug mode:

>>> print "$ bin/buildout\n", system(buildout)
$ bin/buildout
...
Installing myproject.
...
<BLANKLINE>
>>> cat('parts', 'myproject', 'myproject_part_site', 'settings.py')
from myproject_site_config.settings import *
<BLANKLINE>
SECRET_KEY = "..."
<BLANKLINE>
DEBUG = True
TEMPLATE_DEBUG = True
<BLANKLINE>
>>> print system('bin/django diffsettings')
DATABASES = {'default': {'ENGINE': 'django.db.backends.sqlite3', 'NAME': 'storage.db'}}
DEBUG = True
SECRET_KEY = '...'
SETTINGS_MODULE = 'myproject_part_site.settings'  ###
TEMPLATE_DEBUG = True
TIME_ZONE = 'Europe/Rome'

Of course, this is not limited to DEBUG, you can use it for example to override the DATABASES, LOGGING and CACHES settings in the production environment without having to create a whole new settings.py file.

Going production

As we saw above, if our development setup doesn’t differ too much from our production setup (save for the fact that we use a real cache, a more complex RDBMS, etc) then we can use settings-override to manage it:

>>> mkdir('var')
>>> mkdir('var', 'log')
>>> write('buildout.cfg', '''
... [buildout]
... parts =
...     myproject
...
... [myproject]
... recipe = djc.recipe2
... settings-override =
...     DATABASES = {
...         'default': {
...             'ENGINE': 'django.db.backends.postgresql_psycopg2',
...             'HOST': 'localhost',
...             'PORT': '5432',
...             'NAME': 'mydb',
...             'USER': 'mydb',
...             'PASSWORD': 'secret'
...         }
...     }
...     CACHES = {
...         'default': {
...             'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
...             'LOCATION': '127.0.0.1:11211',
...         }
...     }
...     LOGGING = {
...         'version': 1,
...         'disable_existing_loggers': True,
...         'root': { 'level': 'WARNING', 'handlers': ['logfile'], },
...         'formatters': {
...             'verbose': {
...                 'format': '%(levelname)s %(asctime)s %(module)s %(process)d %(thread)d %(message)s'
...             },
...         },
...         'handlers': {
...             'logfile': {
...                 'level': 'ERROR',
...                 'class': 'logging.handlers.RotatingFileHandler',
...                 'filename': 'var/log/myproject.log',
...                 'maxBytes': 1024,
...                 'backupCount': 3,
...             },
...             'console': {
...                 'level': 'DEBUG',
...                 'class': 'logging.StreamHandler',
...                 'formatter': 'verbose'
...             }
...         },
...         'loggers': {
...             'django.db.backends': {
...                 'level': 'ERROR',
...                 'handlers': ['console'],
...                 'propagate': False,
...             },
...         },
...     }
... ''')
>>> print "$ bin/buildout\n", system(buildout)
$ bin/buildout
...
Installing myproject.
...
<BLANKLINE>
>>> print system('bin/django diffsettings')
CACHES = ...
DATABASES = ...
LOGGING = ...
SECRET_KEY = '...'
SETTINGS_MODULE = 'myproject_part_site.settings'  ###
TIME_ZONE = 'Europe/Rome'

This is actually a quite complete (albeit basic) production example, and it can still be managed quite well within the buildout.

If we do have more complex cases, however, it’s probably better to use External settings.

Changing the binary name

As we have said before, the name of the generated binary is always django, without any suffix or prefix.

The rational for this choice is the following:

1. Having the script named django and it being the same no matter how you call the buildout part simplifies getting into development a lot (it’s always bin/django runserver after you run the buildout, and you don’t have to go and look how it is named in that particular buildout)

2. Since in production you will just configure your WSGI server to use multiple processes, there are very few reasons to have multiple Django parts in your buildout

But if you really need to have multiple parts, the default behaviour will have one part overwrite the other’s script. That’s when you need to use the manage-py-file option, which allows you to provide a different name (say, django1 and django2) for the manage script.

First we start by copying the settings of our sample project to two ned different locations, myproject1 and myproject2:

>>> copytree(['sites', 'myproject_site_config'],
...          ['sites', 'myproject1_site_config'])
>>> copytree(['sites', 'myproject_site_config'],
...          ['sites', 'myproject2_site_config'])

Then we write a buildout that has two parts, myproject1 and myproject2, and run it:

>>> write('buildout.cfg', '''
... [buildout]
... parts =
...     myproject1
...     myproject2
...
... [myproject1]
... recipe = djc.recipe2
... manage-py-file = django1
...
... [myproject2]
... recipe = djc.recipe2
... manage-py-file = django2
... ''')
>>> print "$ bin/buildout\n", system(buildout)
$ bin/buildout
...
Installing myproject1.
...
Installing myproject2.
...
<BLANKLINE>

And we will see that it has created two distinct scripts:

>>> ls('bin')
-  buildout
-  django1
-  django2

Custom initialization

Sometimes, you need to do some magic before Django loads everything, in order to use certain features.

For example, Pinax, a very well known social site framework based on Django, needs you to perform certain sys.path magic before initialization.

This kind of customization can be done in two ways:

1. By performing those in settings.py

2. By altering the manage script (and the WSGI one, too)

The first choice might look simpler but it actually hides much more complexity than it is initially visible. The latter is better but, since the script is generated by buildout, we cannot simply edit that file.

Before looking at how you actually do it, let’s make a premise: we can divide this initialization stuff in two main groups.

The first and more common group is when you simply need to set an environment variable: while this can be achieved by doing $ MYVAR=value bin/django, it’s not exactly handy in the long run.

And here comes environment-vars to the rescue!

Let’s look at a concrete example: running Django on Google App Engine. Google App Engine requires you to have a GOOGLE_APPENGINE_PROJECT_ROOT environment variable set, or nothing will work.

Therefore, in order to add it we would write our buildout as follows, with a list of variables and values (separated by space) for each environment variable we want to set:

>>> write('buildout.cfg', '''
... [buildout]
... parts =
...     myproject
...
... [myproject]
... recipe = djc.recipe2
... environment-vars =
...     GOOGLE_APPENGINE_PROJECT_ROOT /my/path
... ''')

And after running it, we can see that the script correctly initializes the environment variable:

>>> print "$ bin/buildout\n", system(buildout)
$ bin/buildout
...
Installing myproject.
...
<BLANKLINE>
>>> cat('bin', 'django')
#!...
<BLANKLINE>
...
<BLANKLINE>
import os
os.environ["GOOGLE_APPENGINE_PROJECT_ROOT"] = r"/my/path"
<BLANKLINE>
...
<BLANKLINE>
os.environ['DJANGO_SETTINGS_MODULE'] = "myproject_part_site.settings"
if IS_14_PLUS:
    execute_from_command_line(sys.argv)
else:
    utility = ManagementUtility(sys.argv)
    utility.execute()

For the second case, the initialization option is provided: this allows you to write (in a format similar to doctest) the python code that you need to be executed before Django starts.

Let’s see how we would make sure that Django won’t start at all if 1 != 1:

>>> write('buildout.cfg', '''
... [buildout]
... parts =
...     myproject
...
... [myproject]
... recipe = djc.recipe2
... initialization =
...     >>> if 1 != 1:
...     ...     raise RuntimeError("I can't run on quantum computers")
... ''')
>>> print "$ bin/buildout\n", system(buildout)
$ bin/buildout
...
Installing myproject.
...
<BLANKLINE>
>>> cat('bin', 'django')
#!...
<BLANKLINE>
...
<BLANKLINE>
if 1 != 1:
    raise RuntimeError("I can't run on quantum computers")
<BLANKLINE>
...
<BLANKLINE>
os.environ['DJANGO_SETTINGS_MODULE'] = "myproject_part_site.settings"
if IS_14_PLUS:
    execute_from_command_line(sys.argv)
else:
    utility = ManagementUtility(sys.argv)
    utility.execute()

Media and static

This is a bit of personal preference. When developing upon work started by someone else, I find it utterly irritating that the upload doesn’t work because, after checking out and running the buildout, I did not do $ mkdir media.

Because:

1. I’m getting old and I tend to forget that

2. Sometimes it’s not media, but var/upload/mediafiles or something else (yes, we programmers tend to express creativity in the most inopportune ways)

That’s why I’ve added two options that, while not being on by default, I wish you have turned on (atleast one of them) if I have to work on your buildout.

The options are media-directory and static-directory, and their values are the path to the media root and the static root respectively. When they are set, the buildout will create them if they don’t exist and then append to the settings module the proper MEDIA_ROOT and STATIC_ROOT setting.

Let’s see them in action. First we check that we don’t have any static or media directory:

>>> isdir('media')
False
>>> isdir('static')
False

Then write and run the buildout:

>>> write('buildout.cfg', '''
... [buildout]
... parts =
...     myproject
...
... [myproject]
... recipe = djc.recipe2
... media-directory = media
... static-directory = static
... ''')
>>> print "$ bin/buildout\n", system(buildout)
$ bin/buildout
...
Installing myproject.
...
<BLANKLINE>

And then see that we have the directories and the settings:

>>> isdir('media')
True
>>> isdir('static')
True
>>> print system('bin/django diffsettings')
DATABASES = ...
MEDIA_ROOT = '...'
SECRET_KEY = '...'
SETTINGS_MODULE = 'myproject_part_site.settings'  ###
STATIC_ROOT = '...'
TIME_ZONE = 'Europe/Rome'

Obviously, you do not need to use them together but they can be used independently.

External settings

Sometimes, one file for all the settings just ain’t enough, or it might turn out that settings-override is not quite handy for you.

That’s why this recipe allows you to use as a settings module anything that’s in in sys.path.

For example, suppose we want to put our production settings in a file on its own: we might then create a file named sites/myproject_site_config/production.py and use that as settings module.

First, let’s create the file:

>>> write('sites', 'myproject_site_config', 'production.py', '''
... from .settings import *
... TIME_ZONE = 'Europe/London'
... ''')

Then we tell the buildout to use the module myproject_site_config.production as settings module instead of the default one, through the settings-module option:

>>> write('buildout.cfg', '''
... [buildout]
... parts =
...     myproject
...
... [myproject]
... recipe = djc.recipe2
... settings-module = myproject_site_config.production
... ''')

And we can then run the buildout and see what happened:

>>> print "$ bin/buildout\n", system(buildout)
$ bin/buildout
...
Installing myproject.
...
<BLANKLINE>
>>> print system('bin/django diffsettings')
DATABASES = ...
SECRET_KEY = '...'
SETTINGS_MODULE = 'myproject_part_site.settings'  ###
TIME_ZONE = 'Europe/London'

And as you can see, the changes took effect.

eggs

A list of eggs that the generated scripts must have access to. This typically includes your application eggs and their dependencies, if the latter are not explicited within the setup.py file.

They can be explicited either as a part option:

>>> write('buildout.cfg', '''
... [buildout]
... parts =
...     myproject
...
... [myproject]
... recipe = djc.recipe2
... eggs = django-gravatar2
... ''')
>>> print "$ bin/buildout\n", system(buildout)
$ bin/buildout
...
Installing myproject.
...
<BLANKLINE>
>>> cat('bin', 'django')
#...
<BLANKLINE>
<BLANKLINE>
import sys
sys.path[0:0] = [
    '.../eggs/django_gravatar2-1.0.4-...egg',
    ...
    ]
<BLANKLINE>
...

Or as a buildout option:

>>> write('buildout.cfg', '''
... [buildout]
... eggs = django-gravatar2
... parts =
...     myproject
...
... [myproject]
... recipe = djc.recipe2
... ''')
>>> print "$ bin/buildout\n", system(buildout)
$ bin/buildout
...
Installing myproject.
...
<BLANKLINE>
>>> cat('bin', 'django')
#...
<BLANKLINE>
<BLANKLINE>
import sys
sys.path[0:0] = [
    '.../eggs/django_gravatar2-1.0.4-...egg',
    ...
    ]
<BLANKLINE>
...

Or both, and they will be merged:

>>> write('buildout.cfg', '''
... [buildout]
... eggs = South
... parts =
...     myproject
...
... [myproject]
... recipe = djc.recipe2
... eggs = django-gravatar2
... ''')
>>> print "$ bin/buildout\n", system(buildout)
$ bin/buildout
...
Installing myproject.
...
<BLANKLINE>
>>> cat('bin', 'django')
#...
<BLANKLINE>
<BLANKLINE>
import sys
sys.path[0:0] = [
    '.../eggs/django_gravatar2-1.0.4-...egg',
    '.../eggs/South-0.7.5-...egg',
    ...
    ]
<BLANKLINE>
...

environment-vars

A list of environment variables to set before execution, each separated by newline and in the format VAR_NAME value.

See Custom initialization for an example.

extra-paths

A list of paths, separated by newline, that should be added to sys.path before the code is executed (allowing the discovery of custom modules).

For example:

>>> mkdir('custom_modules')
>>> write('buildout.cfg', '''
... [buildout]
... parts =
...     myproject
...
... [myproject]
... recipe = djc.recipe2
... extra-paths =
...     custom_modules
... ''')
>>> print "$ bin/buildout\n", system(buildout)
$ bin/buildout
...
Installing myproject.
...
<BLANKLINE>
>>> cat('bin', 'django')
#...
<BLANKLINE>
<BLANKLINE>
import sys
sys.path[0:0] = [
    ...
    '.../custom_modules',
    ]
<BLANKLINE>
...

initialization

Python code, to be formatted like a doctest, that is to be executed before any initialization happens.

See Custom initialization for an example.

manage-py-file

The name of the generated manage script in bin.

See Changing the binary name for an example.

settings-file

The name of the generated settings file (the one that’s autogenerated by buildout at each run).

This option can be quite useful to avoid module name clashes:

>>> write('buildout.cfg', '''
... [buildout]
... parts =
...     myproject
...
... [myproject]
... recipe = djc.recipe2
... settings-file = configuration.py
... ''')
>>> print "$ bin/buildout\n", system(buildout)
$ bin/buildout
...
Installing myproject.
...
<BLANKLINE>
>>> print system('bin/django diffsettings')
DATABASES = ...
SECRET_KEY = '...'
SETTINGS_MODULE = 'myproject_part_site.configuration'  ###
TIME_ZONE = 'Europe/Rome'

settings-module

Loads a custom settings module instead of the conventional one.

See External settings for an example.

settings-override

Specifies some settings (as python code) to be appended to the auto-generated settings file and thus overriding the module-defined ones.

See Debug mode for an example.

sites-directory

Changes the default location of the conventional configuration location (normally the sites directory).

It will be appended to sys.path:

>>> copytree(['sites'], ['mysites'])
>>> write('buildout.cfg', '''
... [buildout]
... parts =
...     myproject
...
... [myproject]
... recipe = djc.recipe2
... sites-directory = mysites
... ''')
>>> print "$ bin/buildout\n", system(buildout)
$ bin/buildout
...
Installing myproject.
...
<BLANKLINE>
>>> cat('bin', 'django')
#...
<BLANKLINE>
<BLANKLINE>
import sys
sys.path[0:0] = [
    ...
    '.../mysites',
    ]
<BLANKLINE>
...

static-directory

Sets the location of STATIC_ROOT and creates it if missing.

See Media and static.

media-directory

Same as static-directory for MEDIA_ROOT.

wsgi-file

Changes the name of the file that contains the WSGI application.

The purpose is similar to settings-file:

>>> write('buildout.cfg', '''
... [buildout]
... parts =
...     myproject
...
... [myproject]
... recipe = djc.recipe2
... wsgi-file = wsgiapp.py
... ''')
>>> print "$ bin/buildout\n", system(buildout)
$ bin/buildout
...
Installing myproject.
...
<BLANKLINE>
>>> ls('parts', 'myproject', 'myproject_part_site')
-  __init__.py
-  settings.py
-  wsgiapp.py