SlapOS toolbox.
Project description
slapos.toolbox
SlapOS toolbox contains :
HTTP Cache Checker
Crawl a website and check the HTTP configuration.
Run
python2 cachechecker.py -o result.txt cachechecker.conf
Example of configuration
[web_checker] url = http://example.org working_directory = /tmp/cachechecker # debug_level = info debug_level = debug file_log_path = /tmp/cachechecker.log # http://labs.omniti.com/people/mark/ats_sa/slides.html#slide-18 traffic_server_cache = cHs cSsNfU [header_list] Last-Modified = True Expires = True Vary = Cookie,Authorization,Accept-Encoding Cache-Control = max-age=600, stale-while-revalidate=360000, public [no_header url=.*] Set-Cookie = None [no_header content-type=(image/.*|application/vnd.ms-fontobject|application/x-font-woff|font/truetype)] Vary = None [header content-type=image/.*] Cache-Control = max-age=3600, stale-if-error=360000, stale-while-revalidate=360000, public [header content-type=(text/css|text/plain|application/x-javascript|.*/javascript|application/vnd.ms-fontobject|application/x-font-woff|font/truetype)] Vary = Accept-Encoding Cache-Control = max-age=3600, stale-if-error=360000, stale-while-revalidate=360000, public # Captcha [header url=(.*/contactus$)] Cache-Control = max-age=0, no-cache # # IE PDF Bug [header url=.*format=pdf] Cache-Control = Private [header url=.*format=] Vary = Cookie,Authorization,Accept-Encoding Cache-Control = max-age=600, stale-while-revalidate=360000, public [erp5_extension_list] prohibited_folder_name_list = web_page_module document_module web_site_module prohibited_file_name_list = WebSection_viewAsWeb Base_viewHistory Base_download list
cloudmgr
networkbench
onetimeupload
onetimeupload is an HTTP application which only accepts one file to be uploaded.
shacache
The network cache server is a NoSQL storage with a REST API.
How does it works
_______________
/ \
| |
------>| NETWORKCACHED |
| ----| |<-----
| | \_______________/ |
GET /key | | |
| | File | PUT / <- data
__|__v______ ______|_____
| | | |
| Client | | Client |
|____________| |____________|
Basically, the networkcached archives the files through HTTP PUT method. When a client want to download the file it just need to provide the key value and the server will send a response with the file data.
API: PUT / : parameter: file uploaded Used to upload/modify an entry GET /<key> Return raw content Raise HTTP error (404) if key does not exist
Installation
$ python2.6 setup.py install
Now it is time to create the ‘networkcached.conf’ file in /etc/networkcached.conf directory, using your preferred text editor (gedit, kate, vim.). Follow text shall be put in this file:
[networkcached] host = 127.0.0.1 port = 5001 cache_base_folder = /var/cache/networkcached/
Run the server:
# networkcached networkcached.conf
Setup Develoment Environment
$ mkdir -p ~/networkcached/downloads $ cd ~/networkcached
Now it is time to create ‘buildout.cfg’ file in ~/networkcached directory, using your preferred text editor (gedit, kate, vim.). Follow text shall be put in this file:
[buildout] extensions = mr.developer auto-checkout = slapos.tool.networkcached download-cache = /nexedi/buildout-networkcached/downloads eggs-directory = /nexedi/buildout-networkcached/eggs parts = networkcached [sources] slapos.tool.networkcached = svn https://svn.erp5.org/repos/vifib/trunk/utils/slapos.tool.networkcached [networkcached] recipe = zc.recipe.egg eggs = slapos.tool.networkcached
Now you bootstrap the buildout:
$ python -S -c 'import urllib;print urllib.urlopen(\
"http://svn.zope.org/*checkout*/zc.buildout/trunk/bootstrap/bootstrap.py"\
).read()' | python -S -
Run the buildout:
$ bin/buildout -v
Now it is time to create the ‘networkcached.conf’ file in ~/networkcached directory, using your preferred text editor (gedit, kate, vim.). Follow text shall be put in this file:
[networkcached] host = 127.0.0.1 port = 5001 cache_base_folder = ~/networkcached/networkcached-database
Now you can start your networkcached server: $ bin/networkcached networkcached.conf or $ bin/networkcached -d networkcached-database -a 127.0.0.1 -p 5002
webrunner
0.65 (2017-01-23)
checkfeedaspromise: returns more information on failure
monitor: run command with low priority using nice
0.64 (2017-01-31)
monitor: merge status2rss with monitor state generation script in order to have consistent result
generatefeed: remove useless items instead of crashing
0.63 (2017-01-09)
monitor: Allow to change promise timeout from UI
0.62 (2016-12-31)
monitor_config_write: CORS apache conf should be rendered from a slapos template
runner: user’s git information can now contain utf-8 characters
monitor: fix cleanup of monitor promises result folder
0.61 (2016-10-26)
monitor: allow to specify a second promise folder
pubhubsub: new argument allowing to run the executable several times in case of failure
0.60 (2016-10-05)
monitor: write pid to file while processing
monitor: run all promises in a single crontask with timeout of 12 seconds per promises.
0.59 (2016-09-16)
runner: add tests
Add scripts to generate RSS feeds and to create promises on RSS feeds
notifier: new arguments to generate verobose logging
0.58 (2016-09-07)
monitor: multiple fixes and improvements
resilient: fix tests
agent: fix console script
0.57 (2016-08-09)
monitor: Included monitor code with unit tests.
0.56 (2016-07-25)
slaprunner: remove create user at first launch, clone repository can be done by slaprunner at startup.
Remove slapos.builder and slapos.monitor as it only contains unused and dead code.
0.55 (2016-03-01)
Switch to mysqlclient, that is a maintained fork of MySQL-python.
Add an option in is-process-older-than-dependency-set to kill the process if older than dependency set.
0.54 (2016-02-25)
Added is-process-older-than-dependency-set script.
0.53 (2015-11-26)
networkbench: Minor fixes
0.52 (2015-09-02)
testagent: Fully reimplementation
networkbench: Accept external configurations and split file logs
0.51 (2015-08-17)
networkbench: Include ping6 and TTFB support
0.50 (2015-07-20)
slaprunner: Allow to configure the window title through ‘instance-name’ parameter.
0.49 (2015-07-17)
slaprunner: filter out useless file extensions in treeview
0.48 (2015-04-09)
slaprunner: Support command exiting with non 0 status in minishell
0.47.3 (2015-02-17)
resiliency tests: hide output of slapos node instance.
equeue: move takeover detection in thread lock.
add erp5 resiliency test suite.
harcode slaprunner request to use dev branch for erp5.
slaprunner resiliency test suite: compatibility with python 2.7.9.
resiliency test suite: do takeover from web interface.
0.47.2 (2015-02-14)
equeue: prevent to run import script if takeover-triggered file exists.
Advertise development of new version.
0.47.1 (2015-02-05)
equeue: unlock lockfile at boot.
0.47 (2015-02-05)
runner: fix syntax when stopping all processes.
equeue: realtime logging.
0.46.1 (2015-01-30)
runner: enhance software/instance explanation to user.
runner: do not remove whole instance root directory, but only instance home directories.
slaprunner resilient test: update API about slapgridResult.
0.46.0 (2015-01-16)
slaprunner: don’t shutdown supervisord.
slaprunner: Simplify process management and make sure ‘process killer’ handler works.
webrunner: add getConnectionParameter API.
equeue: create a lockfile when running importer command.
Equeue: do not register command in database if it was not successful.
0.45.3 (2015-01-12)
slaprunner: Quick and dirty support for ace editor autocompletion
0.45.2 (2014-12-02)
resiliencytest framework: adapt abstract test suite to work with a single clone.
resiliency test suites: send log to erp5 master.
KVM resiliency test suite: use test framework
0.45.1 (2014-11-24)
Fix logrotate and introduce a delay for networkbench.
0.45.0 (2014-11-24)
Resiliency tests: allow to use from inside of a classical erp5testnode.
0.44.0 (2014-11-22)
Add new networkbench command.
0.43.0 (2014-11-13)
Add new slapos-kill command, deprecating killpidfromfile
Make pycurl egg an optional dependency
0.42.0 (2014-10-28)
The egg provides check-web-page-http-cache-hit script to check if a web page is well cached by Squid or TrafficServer.
0.41.0 (2014-10-20)
The egg provides is-local-tcp-port-opened script to check if an ip address + port are currently opened on the local machine by reading in /proc/net/tcp and /proc/net/tcp6.
0.40.4 (2014-10-14)
The equeue server now supports additional parameters to the callbacks, separated by 0 in the JSON protocol. The pubsubnotifier accepts a –transaction-id parameter. If missing, the current timestamp is used. The pubsubnotifier then provides the transaction-id value in the POST call to /notify.
Together, these three changes allow us to put in the crontab:
0 0 * * * /srv/slapgrid/slappartXX/bin/exporter –transaction-id date +%s
and the same timestamp value is reported as transaction id in the equeue logs for BOTH the pull and push operations.
0.40.3 (2014-10-13)
Slaprunner: stop proxy when removing instances content. This fixes the “partition11 table does not exist” error.
0.40.2 (2014-09-25)
Slaprunner: Various bug fix
Slaprunner: Update supervisorctl interaction to work with latest slapos.core revision
0.40.1 (2014-09-02)
Slaprunner: Fix a bad behaviour within slapos
0.40 (2014-09-01)
Slaprunner: Supervisord runs slapos commands
Slaprunner: fixes for git repositories management
0.39.5 (Unreleased)
Slaprunner: Add slave support test.
0.39.4 (2014-08-11)
Slaprunner: Add a minishell feature, using far less bandwith than shellinabox
Slaprunner: Slapgrid state is correctly rendered while compiling
Slaprunner: consider new parameter’s values without needing to restart [719da31a]
Slaprunner: slapgrid’s last result is always accessible through a stored file [a0bc26e9]
Slaprunner: Fix dangerous bug concerning custom parameters [50ca3386]
Slaprunner: various fixes and improvements
0.39.3 (2014-07-11)
Slaprunner: Add folder to favorites [Alain T.]
Slaprunner: Fix https git clone not possible if no login [Alain T.]
Slaprunner: Improve editor full window [Alain T.]
Pubsub: More explicit RSS displayed in reversed chronoligical order [Cedric LN]
Pubsub-notifier: add logger [Cedric LN]
Slaprunner: Various fixes
0.39.2 (2014-05-16)
Slaprunner: Add tab support [Alain T.]
Slaprunner: Add upload file [Alain T.]
Slaprunner: Improve editor full window [Alain T.]
Slaprunner: Various fixes
0.39.1 (2014-04-30)
Slaprunner: Various fixes
0.39 (2014-02-20)
Slaprunner: new web interface design
Slaprunner: one function handle both “run software” and “run instance” [9c660c0]
Slaprunner: building and deploying can be customized [0db1f6b, b33bd1f]
Slaprunner: adds a multi-user feature [efad6d]
Slaprunner: add fullscreen mode for text edition [83d1dc]
Slaprunner: direct access to monitoring of running instance, if it exists [f8e7bf3]
0.38.1 (2013-12-06)
Slaprunner: do not delete proxy.db on each run software [71777fc0]
0.38 (2013-12-03)
Slaprunner: adds an integrated shell [ca6a670a]
Slaprunner: uses basic authentification [05913751]
Slaprunner: adds automated deployment of a Software Release [c8ab1273]
Slaprunner: flask development server replaced by Gunicorn, a WSGI server [48d60d0f]
Slaprunner: new test scenario for auto-deployment [c6007954]
Runner resiliencytestsuite: adds basic auth support [3c03f12b]
Runner resiliencytestsuite: tests can be done on only one Slapos node [07198d87]
0.37.4 (2013-10-15)
Improve QEMU QMP wrapper by adding drive-backup method and other helpers. [0afb7d6, 95d0c8b]
0.37.3 (2013-10-10)
pubsub: don’t swallow output of subprocess to allow debug. [c503484]
0.37.2 (2013-10-10)
Add QEMU QMP wrapper. [9e819a8]
KVM resiliency test: update docstring about how to setup disk image. [dbe347f]
KVM resiliency test: change key for each clone. [7ef1db3]
0.37.1 (2013-10-03)
pubsub notifier: handle timeout and other connection errors. [ac4c75c]
equeue: cast str(timestamp) to please gdbm. [8b067d6]
0.37 (2013-09-30)
equeue: log output of subprocess. [1694937]
slaprunner: don’t send 200 when login is bad. [4a8e10bf]
Improve reliability of resiliency tests.
0.36 (2013-09-05)
Add resiliency test system.
Add kvm resiliency test suite.
Add slaprunner resiliency test suite.
slaprunner: don’t tell user false assumptions [71dc4a08e].
slaprunner: allow profile/instance to be run by GET [4b454814a].
slaprunner: remove boring username/password length check [fcce0336c].
0.35.1 (2013-08-05)
slaprunner: add support for reverse proxy [6c379644].
0.35.0 (2013-07-08)
Resiliency: equeue is rewriten extending SocketServer.ThreadingUnixStreamServer (Thanks to Julien M.). So far outputs of commands invoked by equeue are redirected to /dev/null to avoid locking the whole process. [Cedric Le Ninivin]
Add htpasswd script. This script is meant to produce htpasswd file without installing apache. [Cedric Le Ninivin]
slaprunner: Grammar improvements.
0.34.0 (2013-03-28)
runner: Log textarea is resizable. [Cedric de Saint Martin]
runner: Fix bug where slaproxy hangs: Redirect slapproxy output to stdout. [Cedric de Saint Martin]
lampconfigure: Add support for running SQL scripts in lampconfigure. [Cedric de Saint Martin]
runner: software_type now survives to slaprunner reboot. [Cedric de Saint Martin]
0.33.1 (2013-02-12)
runner: Add default values for git repository while cloning. [Cedric de Saint Martin]
runner: Don’t remove git repo if error occurred during clone. [Cedric de Saint Martin]
runner: Fix bug where software_type of main instance is impossible to set reliably. [Cedric de Saint Martin]
0.33 (2013-01-24)
agent: run tests in order. [Marco Mariani]
slaprunner: Add process killer killing all processes when instance is stopped. [Cedric Le Ninivin]
slaprunner: slapgrid-cp will be run at startup. [Cedric Le Ninivin]
slaprunner: Improved design. [Alain Takoudjou]
slaprunner: Don’t set arbitrary limitations on the complexity of password. [Cedric de Saint Martin]
0.32 (2012-12-13)
pubsub: support multiple notifications and callbacks. [Marco Mariani]
pubsub: print/return errors from subprocess or notifications. [Marco Mariani]
agent: ignore SR status, retrieve instance status with .getStatus(). [Marco Mariani]
slapmonitor: added slapmonitor-xml function + entry point. [Marco Mariani]
0.31.1 (2012-10-02)
slaprunner: Add new file manager. [Alain Takoudjou]
slaprunner: Ask confirm before delete important folder. [Alain Takoudjou]
0.31 (2012-10-02)
slaprunner: Add correct support for software_type. [Cedric de Saint Martin]
0.30 (2012-08-29)
Add promise support on slapcontainer script. [Antoine Catton]
0.29.1 (2012-08-23)
Minor fix on slapcontainer. [Antoine Catton]
0.29 (2012-08-22)
Add slapcontainer stateless script. [Antoine Catton]
0.28.2 (2012-08-17)
Slaprunner: Move password, process pid and config files to etc_dir [Alain Takoudjou]
0.28.1 (2012-08-15)
Update ACE to 08.11.2012 and JQuery to 1.8.0. [Cedric de Saint Martin]
0.28 (2012-08-09)
slaprunner: add support for authentication management. [Alain Takoudjou]
0.27 (2012-08-07)
pubsub: add support for multiple notification servers. [Timothée Lacroix]
0.26.1 (2012-07-09)
runner: Add support to slapos.core 0.26.2 [Cedric de Saint Martin]
0.26 (2012-07-06)
slaprunner now supports authentication and instance parameters. [Alain Takoudjou]
Redisign of Test Agent so that it supports proper instantiation test and report. [Vincent Pelletier]
0.25 (2012-06-19)
slapmonitor is fully functional. [Mohamadou Mbengue]
0.24 (2012-04-19)
Slap Test Agent : test software release installation and instance allocation [Yingjie Xu]
0.22 (2012-04-05)
Install argparse if missing [Łukasz Nowak]
0.21 (2012-04-04)
zodbpack : simple tool to pack ZODB [Łukasz Nowak]
0.20 (2012-03-28)
slaprunner : fix recursive include of .gif files [Alain Takoudjou]
0.19 (2012-03-28)
slaprunner : Add Support for more instances at the same time. [Alain Takoudjou]
slaprunner : Add slapgrid process control and management. [Alain Takoudjou]
slaprunner : Add Cloud9 IDE. [Alain Takoudjou]
slaprunner : Add develop location of cloned git repository. [Alain Takoudjou]
0.18 (2012-02-23)
slaprunner : Add git commit and push support. [Alain Takoudjou]
slaprunner : Solve UI problems in project management. [Alain Takoudjou]
slaprunner : Solve bug where ACE modifies carrier returns. [Alain Takoudjou]
slaprunner : Add ajax view of build log. [Alain Takoudjou]
slaprunner : Allow to edit any file in repository. [Alain Takoudjou]
0.17 (2012-02-10)
slapbuilder: Update slapbuilder to new OpenSuse 12.1. [Cedric Le Ninivin]
slapbuilder: Added automatic IPv6 support thanks to ViFiB OpenVPN. [Cedric Le Ninivin]
slapbuilder: Added virtual machine support. [Cedric Le Ninivin]
slapbuilder: Added “One Disk” feature. [Cedric Le Ninivin]
0.16 (2012-01-20)
Fix MANIFEST.in [Cedric de Saint Martin]
0.15 (2012-01-20)
slaprunner: Adding project and git repository management [Alain Takoudjou]
0.14 (2012-01-09)
onetimeupload: SSL should be provided by an http frontend [Romain Courteaud]
0.13 (2011-12-30)
builder: fix usage of mkstemp [Romain Courteaud]
0.12 (2011-12-29)
builder: dynamic offset calculation [Romain Courteaud]
0.11 (2011-12-23)
equeue: Add execution queue server. [Antoine Catton]
pubsub: Add pubsub server using pubsubhubbub for notifications. [Antoine Catton]
0.10 (2011-11-07)
slapbuilder: Support banging. [Łukasz Nowak]
lamp : Fix bug when using file_token option [Alain Takoudjou]
0.9 (2011-10-29)
Add LAMP tool to manage lamp instances. [Alain Takoudjou]
Add onetimedownload : a simple script which download any http resource and save it into a specified file_path. [Nicolas Delaby]
0.8 (2011-10-24)
Add new style to runner. [Alain Takoudjou]
0.7 (2011-10-17)
generic: Added systool with killpidfromfile. [Łukasz Nowak]
slaprunner: Avoid error when trying to remove instance directory containing not writable directories. [Alain Takoudjou]
0.6 (2011-07-13)
slapbuilder: Typo during PKI repository creation fixed. [Łukasz Nowak]
0.5 (2011-07-12)
slapbuilder: Do not play too much with buildout, SlapOS installation is responsible for correctly featured zc.buildout [Łukasz Nowak]
0.4 (1011-07-11)
Update slaprunner to run on newer slapos (requires at least 0.10 to work). [Cedric de Saint Martin]
0.3 (2011-06-27)
slapbuilder: use slapos provided networkcached buildout.
slapbuilder: fix network issues on busy networks (“Neighbour table overflow.”)
0.2 (2011-06-10)
In slapbuilder: drop to 100 partitions by default, move out PKI from USB key to real disk. [Łukasz Nowak]
0.1 (2011-05-27)
Created slapos.toolbox from slapos.tool.builder, slapos.tool.runner, slapos.tool.cloudmgr, slapos.tool.ontimeupload and slapos.tool.networkcache. [Łukasz Nowak]
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
