Buildout recipe to install varnish
Project description
Varnish recipe for buildout
plone.recipe.varnish is a zc.buildout recipe to install Varnish. Even though the name contains the name Plone, there is nothing Plone specific about this recipe: it works for non-Zope sites just as well.
Configuring it is very simple. For example:
[varnish-build]
recipe = zc.recipe.cmmi
url = ${varnish:download-url}
[varnish]
recipe = plone.recipe.varnish
daemon = ${buildout:parts-directory}/varnish-build/sbin/varnishd
bind = 127.0.0.1:8000
backends = 127.0.0.1:8080
cache-size = 1G
This configures two buildout parts: varnish-build which will download, compile and install varnish, and varnish which runs Varnish, configured to listen on 127.0.0.1:8000 for requests, using a 1 gigabyte cache and sending requests to a backend at 127.0.0.1:8080.
Wrappers for all the varnish commands are created in the bin directory of your buildout.
Please note that the configuration generated by this recipe requires Varnish 2.0.3 or later. The download-url provided by the recipe points to the latest tested Varnish release compatible with the generated configuration.
Virtual hosting
Varnish supports virtual hosting by selecting a different backend server based on headers on the incoming request. You can configure the backends through the backends option:
[varnish-instance] backends = plone.org:127.0.0.1:8000 plone.net:127.0.0.1:9000
This will generate a configuration which sends all traffic for the plone.org host to a backend server running on port 8000 while all traffic for the plone.net host is send to port 9000.
Zope 2 hosting (with Virtual Host Monster)
If you are using Zope 2 as backend server you will need to rewrite the URL so the Zope Virtual Host Monster (VHM) can generate correct links for links in your pages. This can be done either by a web server such as Apache or nginx (placed either in front or behind Varnish) but can also be done by Varnish itself.
The three options are described below.
Option 1 (rewrites after Varnish)
If generating these VHM-style URLs in a proxy behind Varnish (or if using VHM’s ‘mapping’ feature), no extra Varnish configuration is needed. Just make sure the “backends” option directs the traffic to the proxy.
Option 2 (rewrites before Varnish)
If generating these VHM-style URLs in a proxy in-front of Varnish, no extra Varnish configuration is needed as long as the original hostname is still retained in the URL. If the hostname is not retained, you can tell Varnish to direct requests based on the “path” instead of the hostname. For example:
[varnish-instance] backends = /VirtualHostBase/http/plone.org:80/Plone:127.0.0.1:8000 /VirtualHostBase/http/plone.net:80/Plone:127.0.0.1:9000
This will generate a configuration which sends all traffic for any request whose path starts with “/VirtualHostBase/http/plone.org:80/Plone” to a backend server running at 127.0.0.1 on port 8000, while request paths starting with “/VirtualHostBase/http/plone.net:80/Plone” are sent to port 9000.
Option 3 (rewrites within Varnish)
To have Varnish generate these VHM-style URLs, you can use the zope2_vhm_map option. Here is an example:
[varnish-instance]
zope2_vhm_map =
plone.org:/plone
plone.net:/plone
This tells us that the domain plone.org should be mapped to the location /plone in the backend. By combining this with the information from the backends option a varnish configuration will be generated that maps URLs correctly.
Load Balancing
Varnish supports load balancing by configuring a director for a pool of backends. This director sends the incoming requests that cannot be fulfulled by varnish to backends in the pool in either random or round robin fashion. You can configure the director via the balancer option:
[varnish-instance] balancer = random
This will generate a configuration which sends all traffic to the director, which will choose a ‘random’ backend server to fulfill the request if the content requested is not cached by varnish itself.
plone.recipe.varnish reference
The plone.recipe.varnish recipe create a Varnish configuration file and creates a wrapper script inside your buildout that will start Varnish with the correct configuration.
Please note that the configuration generated by this recipe requires Varnish 2.0.3 or later.
It can be configured with any of these options:
- daemon
The path of the varnish daemon to use. Set this to the sbin/varnishd executable created by the zc.recipe.cmmi build.
- mode
Specify whether the varnish daemon should run in ‘daemon’ or ‘foreground’ mode. The latter is useful when varnish is run by service supervision tools like daemontools or runit. Defaults to ‘daemon’.
- cache-size
The size of the cache (limited to 2G on 32bit systems).
- bind
Hostname and port on which Varnish will listen for requests. Defaults to 127.0.0.1:8000.
- config
Path for a Varnish VCL configuration to use. If you use this option you can not use the backends, zope2_vhm_map or verbose-headers options .
- backends
Specifies the backend or backends which will process the (uncached) requests. The syntax for backends:
[<hostname>][/<path>]:<ip address>:<port>
The optional ‘hostname’ and ‘path’ allows you to do virtual hosting. If multiple backends are specified then each backend must include either a hostname or path (or both) so that Varnish can direct the matching request to the appropriate backend. Defaults to 127.0.0.1:8080.
- zope2_vhm_map
Defines a virtual host mapping for Zope servers. This is a list of hostname:ZODB location entries which specify the location inside Zope where the website for a virtual host lives.
- verbose-headers
Varnish VCL configuration: a http-response header line X-Varnish-Action is set for debugging purposes. It shows a hit, why it bypass/fetch from backend and if if the object was inserted into cache. Possible values: on or off (default).
- telnet
If specified sets the hostname and port on which Varnish will listen for commands using its telnet interface.
- user
The name of the user varnish should switch to before accepting any requests. Defaults to nobody.
- group
The name of the group varnish should switch to before accepting any request. This defaults to the main group for the specified user.
- balancer
If included and set to either ‘random’ or ‘round-robin’, this option configures varnish to load balance the servers specified by the ‘backends’ directive. Possible values: none (default), round-robin or random.
- connect-timeout
If specified, this option configures the connection timeout (in seconds) for Varnish connecting to a backend server. Varnish will only try to connect to a given backend for this many seconds before giving up. Defaults to 0.4s, as per Varnish’s default settings.
- first-byte-timeout
If specified, this option configures the timeout (in seconds) for Varnish receiving the first byte from a backend. Varnish will only wait for this many seconds before giving up. A value of 0s means Varnish will never time out. Defaults to 300s, see http://vincentfretin.ecreall.com/articles/varnish-guru-meditation-on-timeout
- between-bytes-timeout
If specified, this option configures the timeout (in seconds) for Varnish waiting between bytes when receiving data from a backend. Varnish will only wait this many seconds between bytes before giving up. A value of 0s means this will never time out. Defaults to 60s, as per Varnish’s default settings.
- runtime-parameters
Runtime parameter configuration options. The full list of available options can be found in the manpage varnishd(1) for your version of varnish. Examples include ‘thread_pool_max’, ‘thread_pool_min’, ‘sess_timeout’.
Changelog
1.1b1 (2010-04-25)
Updated advertises Varnish to 2.1 and adjusted config. [hannosch]
Correct documentation for the daemon setting and remove the default. [hannosch]
Removed the deprecated build recipe. [hannosch]
Added basic test infrastructure and a test for the simple buildout. [hannosch]
Use the built-in set type instead of the deprecated sets module. This recipe now requires at least Python 2.4. [hannosch]
Added the ability to configure runtime parameters in the varnish runner configuration and added information to the documentation for it. [benliles]
Improve readability of the generated config. [ldr]
1.0.2 (2010-01-18)
Update proposed Varnish to 2.0.6. [hannosch]
Further documentation cleanup. [hannosch, vincentfretin]
1.0.1 (2009-11-27)
Expose the download-url of a known-good Varnish release that works with the configuration produced by the instance recipe. [hannosch]
Consistently use tabs in the generated vcl file. [hannosch]
Whitespace and documentation cleanup. [hannosch]
1.0 (2009-08-27)
Made the vcl template build its acl purge section. At present, the vcl will only allow purges coming from the local host. If we have multiple hosts that are separate from localhost, any PURGE requests will be denied without this. See http://varnish.projects.linpro.no/wiki/VCLExamplePurging [rockdj]
Added ability to set various Varnish timeouts (connect_timeout, first_byte_timeout, and between_bytes_timeout) from each option in the buildout. Default values are set at Varnish defaults of 0.4s for connect_timeout, and 60s for between_bytes_timeout. Time for first_byte_timeout is set at 300s as per plone.recipe.varnish 1.0rc9. [rockdj]
Set req.http.host for incoming virtual hosted URLs. Without setting this, purge requests sent from hosts other than localhost (the only host in the acl purge list) will result in a 404 message. See http://davidjb.com/blog/2009/01/plone-varnish-configuration-cache-hits-purge-fails [rockdj]
1.0rc11 (2009-06-27)
Reintroduced grace options. What the varnish documentation say about grace: “varnish serves stale (but cacheable) objects while retrieving object from backend”. The problem is “default_ttl” value is 120s (see bin/varnishd/mgt_param.c in varnish 2.0.4). Added a special rule for createObject url to not look up in the cache. [vincentfretin]
1.0rc10 (2009-06-26)
1.0rc9 generated broken configuration with balancer=none [vincentfretin]
1.0rc9 (2009-06-25)
Do not set req.grace and obj.grace. See http://vincentfretin.ecreall.com/articles/varnish-user-be-careful [vincentfretin, maurits]
Removed header_hit_deliver and header_hit_notcacheable debug messages from default template. It is not safe to assign to the object during vcl_hit until http://varnish.projects.linpro.no/ticket/310 is not fixed. See also http://kristian.blog.linpro.no/2009/05/25/common-varnish-issues. [hannosch]
Updated to refer to Varnish 2.0.4. Added a first_byte_timeout value of 300 seconds to the backend definitions. This is a new option since Varnish 2.0.3 and by default set to 60 seconds. This is arguably too low for certain edit operations in Plone sites. [hannosch]
1.0rc8 (2008-02-12)
Remove the custom vcl_hash from the template. Adding the Accept-Encoding header to the cache break effectively breaks purging since nobody will ever include those headers in a PURGE request. To make this safe we just remove the Accept-Encoding header from all incoming requests as well. [wichert]
1.0rc7 (2008-11-26)
Be more explicit about deprecating the :build entry point. [wichert]
Make the :instance specifier optional: after :build has been removed we can deprecate :instance as well. [wichert]
1.0rc6 (2008-09-22)
Deprecate plone.recipe.varnish:build in favour of zc.recipe.cmmi: it does not make sense to duplicate its logic here. [wichert]
Add feature to enable verbose headers in varnish.vcl. This is primary interesting for debugging of cache-settings. See README.txt. [jensens]
Deal better with sources which do not have executable-bits set or are svn exports. [wichert]
The 1.0rc5 release was broken and has been retracted. Currently the trunk is only usable with the Varnish 2.0-beta1 and later. [hannosch]
1.0rc5 (2008-04-27)
Pipe is evil: it pipes the whole connection to the backend which means varnish will no longer process any further requests if HTTP pipelining is used. Switch to using pass instead. [wichert]
Add a default_ttl of zero seconds to the Varnish runner to avoid a Varnish bug with the handling of an Expires header with a date in the past. [newbery]
Merged branches/newbery-hostnamepath. [newbery]
We don’t need to include Accept-Encoding in the hash. Varnish takes care of Vary negotiation already. [newbery]
1.0rc4 (2008-03-18)
Fixed typos / whitespace. [hannosch]
Varnish 1.1.2 is out. [wichert]
Merged witsch-foreground-support back to trunk. [witsch]
Use a pidfile. [wichert]
1.0rc3 (2007-09-02)
Fixed a bug where options[“location”] was being used before it was being set. [rocky]
Made the module name determination a little more robust during createVarnishConfig so that recipes that specify version deps still work. [rocky]
Do not use defaults for user and group. [wichert]
We do need the parts: we use it for the file storage. [wichert]
1.0rc2 (2007-08-29)
Add an option to use an existing configuration file. [wichert]
Remove hardcoded caching for images, binaries, CSS and javascript. This should be done by the backend server or a custom varnish configuration. [wichert]
Add Accept-Encoding to the cache key so we can handle compressed content. [wichert]
Test if a bin-directory exists. This allows us to compile varnish 1.0 which does not have an sbin directory. [wichert]
1.0rc1 (2007-08-27)
Document the OSX bugfix we apply when building varnish. [wichert]
Add a dummy update method to prevent needless recompiles. [wichert]
Update for Varnish 1.1.1. [wichert]
1.0b2 (2007-08-25)
When building from svn, we need to run autogen.sh. [optilude]
Refactor the recipe: there are now separate recipes to build and configure Varnish. This makes it possible to reconfigure varnish without having to recompile with as well as using an already installed varnish. [wichert]
Move the OSX patching code into a separate method. [wichert]
Use pass for non-GET/HEAD requests. This makes a bit more sense and fixes a login problem for Plone sites. [wichert]
Reorganize a bit for readability. [wichert]
Support Python 2.3 as well. [wichert]
Make it possible to specify the user and group as well. [wichert]
Do not create the source directory - we move the extracted source in its place later. [wichert]
If running on OS X, patch libtool as described in http://varnish.projects.linpro.no/ticket/118 and http://thread.gmane.org/gmane.comp.web.varnish.misc/668/focus=669. [optilude]
VCL is not C. You need the curlies even on single-line if statements. [optilude]
This rewriting style only works on Zope 3 - Zope 3 reinvented that wheel. [wichert]
Add support for If-Modified-Since and If-None-Match requests. Thanks to newbery for the suggstions. [wichert]
Explicitly mention that there is nothing Plone or Zope specific about this recipe. [wichert]
1.0b1 (2007-08-04)
More documentation. [wichert]
Ignore the port information in the host header. [wichert]
Use the port varnish is bound to in the VHM mapping. [wichert]
Define all default values centrally. [wichert]
Add support for Zope virtual hosts. [wichert]
Add support for virtual hosting. [wichert]
Initial import of Varnish recipe. [wichert]
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
