Instant code reloading for Plone using a fork loop
Project description
sauna.reload: so that you can finish your Plone development today and relax in sauna after calling it a day
Introduction
sauna.reload is a developer tool which restarts Plone and reloads your changed source code every time you save a file. The restart is optimized for speed and happens much faster than a normal start up process.
Edit your code
Save
Go to a browser and hit Refresh ➔ your latest changes are active
It greatly simplifies your Plone development workflow and gives back the agility of Python.
It works with any code.
sauna.reload works on OSX and Linux with Plone 4.0 and 4.1. In theory works on Windows, but no one has looked into that yet.
User comments
“I don’t want to use sauna.reload as I can knit a row while I restart …”
“no more do I start a 5 minute cigarette every time Plone restarts for 30 seconds… ok wait, this kind of joy leads to poetry, I’m gonna stop here.”
Installation
Here are brief installation instructions.
Prerequisitements
In order to take the advantage of sauna.reload
You know how to develop your own Plone add-ons and basics of buildout folder structure
You know UNIX command line basics
You know how to run buildout
You are running Linux or OSX operating system
No knowledge for warming up sauna is needed in order to use this product.
Installing buildout configuration
The recommended installation configuration is to use ZEO server + 1 client for the development. This method is valid for Plone 4.1 and higher. You can use sauna.reload without separate ZEO database server, using instance command, but in this case you’ll risk database corruption on restarts.
Below is the recommended approach how to enable sauna.reload for your development environment. However, since this product is only for development use the data loss should not be a big deal.
Add sauna.reload to your buildout.cfg file:
buildout.cfg:
[buildout]
eggs +=
sauna.reload
# This section is either client1 / instance depending
# on your buildout
[instance]
zope-conf-additional = %import sauna.reload
OSX notes
If you are using vim (or macvim) on OSX, you must disable vim’s writebackups to allow WatchDog to see your modifications (otherwise vim will technicallyj create a new file on each save and WatchDog doesn’t report the modification back to sauna.reload).
So, Add the following to the end of your .vimrc:
set noswapfile set nobackup set nowritebackup
Similar issues have been reported with some other OSX-editors. Tips and fixes for these are welcome.
Ubuntu / Debian / Linux notes
You might need to raise your open files ulimit if you are operating on the large set of files, both hard and soft limit.
104000 is a known good value.
If your ulimit is too low you’ll get very misleading OSError: No space left on device.
Usage: start Plone in reload enabled manner
To start Plone with reload functionality you need to give special environment variable RELOAD_PATH for your client1 command:
# Start database server bin/zeoserver start # Start Plone with reload enabled RELOAD_PATH=src bin/client1 fg
Or if you want to optimize load speed you can directly specify only some of your development products:
RELOAD_PATH=src/my.product:src/my.another.product bin/client1 fg
When reload is active you should see something like this in your console when Zope starts up:
2011-08-10 13:28:59 INFO sauna.reload Starting file monitor on /Users/moo/code/x/plone4/src 2011-08-10 13:29:02 INFO sauna.reload We saved at least 29.8229699135 seconds from boot up time 2011-08-10 13:29:02 INFO sauna.reload Overview available at: http://127.0.0.1:8080/@@saunareload 2011-08-10 13:29:02 INFO sauna.reload Fork loop starting on process 14607 2011-08-10 13:29:02 INFO sauna.reload Booted up new new child in 0.104816913605 seconds. Pid 14608
… and when you save some file in src folder:
2011-08-10 13:29:41 INFO SignalHandler Caught signal SIGINT 2011-08-10 13:29:41 INFO Z2 Shutting down 2011-08-10 13:29:42 INFO SignalHandler Caught signal SIGCHLD 2011-08-10 13:29:42 INFO sauna.reload Booted up new new child in 0.123936891556 seconds. Pid 14609
CTRL+C should terminate Zope normally. There might be stil some kinks and error messages with shutdown.
Only eggs loaded through z3c.autoinclude can be reloaded. Make sure you don’t use buildout.cfg zcml = directive for your eggs or sauna.reload silently ignores changes.
Manual reload
There is also a view on Zope2 root from which it is possible to manually reload code:
http://127.0.0.1:8080/@@saunareload
Debugging with sauna.reload
Regular import pdb; pdb.set_trace() will work just fine with sauna.reload and using ipdb as a drop-in for pdb will work fine as well. When reloads happen while in either pdb or ipdb, the debugger will get killed. To avoid losing your terminal echo, because of reload unexpectedly killing your debugger, you may add the following to your ~/.pdbrc:
import termios, sys term_fd = sys.stdin.fileno() term_echo = termios.tcgetattr(term_fd) term_echo[3] = term_echo[3] | termios.ECHO term_result = termios.tcsetattr(term_fd, termios.TCSADRAIN, term_echo)
As ipdb extends pdb, this configuration file will also work to restore the terminal echo.
sauna.reload also should work nicely with PdbTextMateSupport and PdbSublimeTextSupport. Unfortunately, we haven’t seen it working with vimpdb yet.
Background
sauna.reload is an attempt to recreate plone.reload without the issues it has. Like being unable to reload new grokked views or portlet code. This project was started on Plone Sauna Sprint 2011. There for the name, sauna.reload.
sauna.reload does reloading by using a fork loop. So actually it does not reload the code, but restarts small part of Zope2. That’s why it can it reload stuff plone.reload cannot.
It does following on Zope2 startup:
Defers loading of your development packages by hooking into PEP 302 loader and changing their z3c.autoinclude target module (and monkeypatching fiveconfigure/metaconfigure for legacy packages).
Starts a watcher thread which monitors changes in your development py-files
Stops loading of Zope2 in zope.processlifetime.IProcessStarting event by stepping into a infinite loop; Just before this, tries to load all non-developed dependencies of your development packages (resolved by z3c.autoinclude)
It forks a new child and lets it pass the loop
Loads all your development packages invoking z3c.autoinclude (and fiveconfigure/metaconfigure for legacy packages). This is fast!
And now every time when the watcher thread detects a change in development files it will signal the child to shutdown and the child will signal the parent to fork a new child when it is just about to close itself
Just before dying, the child saves Data.fs.index to help the new child to see the changes in ZODB (by loading the saved index)
GOTO 4
Internally sauna.reload uses WatchDog Python component for monitoring file-system change events.
See also Ruby guys on fork trick.
Events
sauna.reload emits couple of events during reloading.
- sauna.reload.events.INewChildForked
Emited immediately after new process is forked. No development packages have been yet installed. Useful if you want to do something before your code gets loaded. Note that you cannot listen this event on a package that is marked for reloading as it is not yet installed when this is fired.
- sauna.reload.events.INewChildIsReady
Emitted when all the development packages has been installed to the new forked child. Useful for notifications etc.
Limitations
sauna.reload supports only Plone >= 4.0 for FileStorage and Plone >= 4.1 for ZEO ClientStorage.
Does not handle dependencies
sauna.reload has a major pitfall. Because it depends on deferring loading of packages to be watched and reloaded, also every package depending on those packages should be defined to be reloaded (in RELOAD_PATH). And sauna.reload doesn’t resolve those dependencies automatically!
Forces loading all dependencies
An another potential troublemaker is that sauna.reload performs implicit <includeDependencies package="." /> for every package in RELOAD_PATH (to preload dependencies for those packages to speed up the reload).
This may break up some packages.
Does not work with core packages
We are sorry that sauna.reload may not work for everyone. For example, reloading of core Plone packages could be tricky, if not impossible, because many of them are explicitly included by configure.zcml of CMFPlone and are not using z3c.autoinclude at all. You would have to remove the dependency from CMFPlone for development to make it work…
Product installation order is altered
Also because the product installation order is altered (by all the above) you may find some issue if your product does something funky on installation or at import time.
Please report any other issues at Github issue tracker.
Troubleshooting
Report all issues on GitHub.
My code does not reload properly
You’ll see reload process going on in the terminal, but your code is still not loaded.
You should see following warnings with zcml-paths from your products:
2011-08-13 09:38:12 ERROR sauna.reload.child Cannot reload src/sauna.reload/sauna/reload/configure.zcml.
Make sure your code is hooked into Plone through z3c.autoinclude and NOT using explicit zcml = directive in buildout.cfg.
Retrofit your eggs with autoinclude support if needed
Remove zcml = lines for your eggs in buildout.cfg
Rerun buildout (remember bin/buildout -c development.cfg)
Restart Plone with sauna.reload enabled
I want to exclude my meta.zcml from reload
It’s possible to manually exclude configuration files from reloading by forcing them to be loaded before forkloop in a custom site.zcml. Be aware, that when site-zcml option is used, zope2instance ignores zcml and zcml-additional options.
Define a custom site.zcml in your buildout.cfg with:
[instance]
recipe = plone.recipe.zope2instance
...
site-zcml =
<configure xmlns="http://namespaces.zope.org/zope"
xmlns:meta="http://namespaces.zope.org/meta"
xmlns:five="http://namespaces.zope.org/five">
<include package="Products.Five" />
<meta:redefinePermission from="zope2.Public" to="zope.Public" />
<five:loadProducts file="meta.zcml"/>
<!-- Add include for your package's meta.zcml here: -->
<include package="my.product" file="meta.zcml" />
<five:loadProducts />
<five:loadProductsOverrides />
<securityPolicy component="Products.Five.security.FiveSecurityPolicy" />
</configure>
I want to exclude ALL meta.zcml from reload
Sure. See the tip above and use the snippet below instead:
[instance]
recipe = plone.recipe.zope2instance
...
site-zcml =
<configure xmlns="http://namespaces.zope.org/zope"
xmlns:meta="http://namespaces.zope.org/meta"
xmlns:five="http://namespaces.zope.org/five">
<include package="Products.Five" />
<meta:redefinePermission from="zope2.Public" to="zope.Public" />
<five:loadProducts file="meta.zcml"/>
<!-- Add autoinclude-directive for deferred meta.zcml here: -->
<includePlugins package="sauna.reload" file="meta.zcml" />
<five:loadProducts />
<five:loadProductsOverrides />
<securityPolicy component="Products.Five.security.FiveSecurityPolicy" />
</configure>
sauna.reload is not active - nothing printed on console
Check that your buildout.cfg includes zope-conf-additional line.
If using separate development.cfg make sure you run your buildout using it:
bin/buildout -c development.cfg
Source
On GitHub.
Credits
Esa-Matti Suuronen [esa-matti aet suuronen.org]
Asko Soukka [asko.soukka aet iki.fi]
Mikko Ohtamaa (idea, doccing)
Vilmos Somogyi (logo). The logo was originally the logo of Sauna Sprint 2011 and it was created by Vilmos Somogyi.
Martijn Pieters for teaching us PEP 302 -loader trick at Sauna Sprint 2011.
Yesudeep Mangalapilly for creating WatchDog component and providing support for Sauna Sprint team using it
Thanks to all happy hackers on Sauna Sprint 2011!
300 kg of beer was consumed to create this package (at least). Also several kilos of firewood, one axe, one chainsaw and one boat.
We still need testers and contributors. You are very welcome!
Changelog
0.5.4 (2016-02-24)
Add workaround for watchdog raising annoying KeyErrors [datakurre]
0.5.3 (2014-10-19)
Fix issue where in Zope version >= 2.13.19 the storage is explicitly closed on shutdown. This caused an attribute error on reload, since the _rpc_mgr was None. Test for is_connected before closing connection and cache. [sunew]
0.5.2 (2014-10-18)
Log ‘cannot reload’ as warning instead of error to avoid triggering postmortem debugging [sunew]
0.5.1 (2013-02-20)
Fix IOError when sauna.reload is imported for a forked process [datakurre]
0.5.0 (2012-12-28)
Prepopulate platform.uname, before it gets lost in the stack. All calls using platform.uname() values would fail with an IOError(10). [esteele]
Fixed reloadings of Products-namespace eggs on Plone 4.1 and Zope 2.13 [miohtama]
Log out the reason what killed the child [miohtama]
0.4.3 (2012-04-25)
Updated README about ZEO-client support to be limited to Plone >= 4.1.
0.4.2 (2012-04-24)
Fixed BlogStorage-support (regression from 0.3.3). [datakurre]
0.4.0 (2012-04-22)
Added log-messages about successfully reloaded configuration files. Closes https://github.com/epeli/sauna.reload/issues/11 [datakurre]
Added support for reloading a ZEO-client on Plone >= 4.1. Fixes https://github.com/collective/sauna.reload/issues/1 [datakurre]
Added support to p.a.themingplugins when p.a.theming is installed. Fixes https://github.com/epeli/sauna.reload/issues/15 [datakurre]
Fixed an issue of zope2.Public not mapped to zope.Public when confiuring products during reload. Fixes https://github.com/collective/sauna.reload/issues/2 [datakurre]
Fixed to depend on watchdog >= 0.6.0 to support OSX out-of-the-box. [datakurre]
0.3.3 (2011-10-17)
Fixed an issue in initializing more than one reloaded Archetype products.
0.3.2 (2011-08-21)
Fixed to work better with PdbSublimeText/TextMateSupport.
README-updates (mainly for exluding zcmls from reload).
0.3.1 – 2011-08-19
Support for Five-initialized products (e.g. Archetypes) on Plone 4.1.
Licensed under ZPL.
0.3.0 (2011-08-12)
Support Plone 4.1
Heavily edited readme [miohtama]
Show nice error log message if product installation failed to defer ie. is not reloadable.
0.2.1 (2011-08-03)
Add INewChildIsReady event and move INewChildForked event
Remove overly used prints and use logging
Can now reload __init__.py from package root
0.2.0 (2011-08-04)
Support for Five-initialized products (e.g. Archetypes).
Browser View for manual reloading
Emit INewChildForked events
Many bug fixes
0.1.1 (2011-08-03)
Added missing egg description.
Added forgotten reload for localizations.
Prefixed commands in README with $.
0.1.0 (2011-08-03)
First experimental release.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
