slapos.toolbox 0.129

slapos.toolbox

SlapOS toolbox contains :

HTTP Cache Checker

Crawl a website and check the HTTP configuration.

Run

python2 cachechecker.py -o result.txt cachechecker.conf

Example of configuration

[web_checker]
url = http://example.org

working_directory = /tmp/cachechecker

# debug_level = info
debug_level = debug
file_log_path = /tmp/cachechecker.log

# http://labs.omniti.com/people/mark/ats_sa/slides.html#slide-18
traffic_server_cache = cHs
  cSsNfU

[header_list]
Last-Modified = True
Expires = True
Vary = Cookie,Authorization,Accept-Encoding
Cache-Control = max-age=600, stale-while-revalidate=360000, public

[no_header url=.*]
Set-Cookie = None

[no_header content-type=(image/.*|application/vnd.ms-fontobject|application/x-font-woff|font/truetype)]
Vary = None

[header content-type=image/.*]
Cache-Control = max-age=3600, stale-if-error=360000, stale-while-revalidate=360000, public

[header content-type=(text/css|text/plain|application/x-javascript|.*/javascript|application/vnd.ms-fontobject|application/x-font-woff|font/truetype)]
Vary = Accept-Encoding
Cache-Control = max-age=3600, stale-if-error=360000, stale-while-revalidate=360000, public

# Captcha
[header url=(.*/contactus$)]
Cache-Control = max-age=0, no-cache

# # IE PDF Bug
[header url=.*format=pdf]
Cache-Control = Private

[header url=.*format=]
Vary = Cookie,Authorization,Accept-Encoding
Cache-Control = max-age=600, stale-while-revalidate=360000, public

[erp5_extension_list]
prohibited_folder_name_list = web_page_module
  document_module
  web_site_module
prohibited_file_name_list = WebSection_viewAsWeb
  Base_viewHistory
  Base_download
  list

networkbench

onetimeupload

onetimeupload is an HTTP application which only accepts one file to be uploaded.

webrunner

0.129 (2023-01-18)

• promise/plugin: added alarms for node monitoring: check_ram_usage, check_network_errors_packets, check_network_transit, check_cpu_temperature

• monitor: use gmtime to query data in collect db

• resiliencytestsuite,kvm: python3 fixes

0.128 (2022-06-30)

• notifier: support Python 3.8+

0.127 (2022-06-05)

• qemuqmpclient: support qemu 7+

0.126 (2021-10-04)

• plugin/check_file_state: fail on first problem

0.125 (2021-09-30)

• plugin/check_file_state: fix empty url parameter usage

• plugin/check_file_state: allow to assert absence of the file

0.124 (2021-08-25)

• pubsub: Fix Python3 incompatibilities

0.123 (2021-07-23)

• plugin/check_url_available: many improvements, now supports basic auth

  WARNING, backward incompatible changes:

  • http_code option is now http-code

  • check-secure option has been removed. Users should specify http-code = 401 instead.

0.122 (2021-06-29)

• promise/plugin: Fix check_socket_listening promise

0.121 (2021-06-24)

• qemuqmpclient: Support qemu 5.x+

0.120 (2021-06-23)

• promise/plugin: Add check_socket_listening promise

0.119 (2021-04-21)

• runner: python3 fixes for the web interface

0.118 (2021-03-29)

• dnsresolver: new tool for massive DNS queries

0.117 (2021-02-03)

• promise/plugin: Implement check_service_state promise plugin

0.116 (2021-02-01)

• runner: import werkzeug.middleware.proxy_fix.ProxyFix from its new location (for compatibility with werkzeug 1.0.x)

0.115 (2021-01-27)

• check_surykatka_json: minimise messages to avoid information bloat

0.114 (2021-01-22)

• apachedex: expect apachedex options to be passed as a file

• check_free_disk_space: Btrfs has no fixed inode limit

0.113 (2020-11-11)

• apachedex: fix argument parsing

0.112 (2020-10-20)

• test: fix for random failures

• runner: fix deployment of instance through GUI

0.111 (2020-09-14)

• promise: Support http-header-dict in check_surykatka_json

0.110 (2020-08-25)

• promise: Fix argument parsing in check_slow_queries_digest_result

• promise: Make check_slow_queries_digest_result use .xz compressed reports, which introduces a dependency to lzma.backports

0.109 (2020-03-12)

• promise: Fix certificate expiration reporting in check_surykatka_json

0.108 (2020-03-02)

• promise: check_certificate promise introduced

0.107 (2020-02-28)

• runner: more Py3 support.

0.106 (2020-02-14)

• promise: Drop UTC-now in check_surykatka_json

0.105 (2020-02-12)

• promise: Support total_seconds in check_surykatka_json

• promise: Support failure-amount in check_surykatka_json

• promise: Bugfixes for check_surykatka_json

0.104 (2020-01-09)

• monitor: Cleanup stale history.json files

0.103 (2020-01-02)

• promise: Fix check_surykatka_json for http-only checks

0.102 (2019-12-31)

• promise: Support SSL checks in check_surykatka_json

0.101 (2019-12-12)

• monitor: the monitor.global.json is under /public directory not /share/public

• promise: Add check_surykatka_json promise

0.100 (2019-12-10)

• monitor: stabilise bootstrap status file creation

0.99 (2019-12-05)

• monitor: support configuration w/o legacy-promise-folder

• promise: consider missing pid file as failure in monitor bootstrap

0.98 (2019-12-02)

• promise: fix threshold-days in check_free_disk_space

0.97 (2019-11-14)

• resiliencytest: Fixes for the test

• equeue: use BytesIO instead of StringiO as buffer

• promise: Add more information on error

• promise: Support threshold-days in check_free_disk_space

• promise: Support simple threshold in check_free_disk_space

0.96 (2019-09-27)

• promise: add check_command_execute

0.95 (2019-09-09)

• add support for Python 3

• remove slapos.cloudmgr

• runner: force instantiation of partitions on manual action

• promise: add check_neo_health for ERP5/NEO SR

0.94 (2019-06-19)

• implement promise.plugin.check_file_state

• promise.plugin.check_url_available: set good default timeout

0.93 (2019-06-13)

• promise.plugin: enhance and improve check_url_available

• monitor: catch timeout rise by ssl.SSLError

0.92 (2019-04-10)

• runner: more robust exporter script (don’t check excluded files, support broken symlinks)

• runner: give more information when exporter script fails

• monitor: escape title in OPML

0.91 (2019-03-28)

• runner: fixed the exporter script

0.90 (2019-03-20)

• fixed wrong upload

0.89 (2019-03-20)

• promise.plugin: Support ATS with traffic_ctl

• resiliencytest: remove some slapparts hardcoded in test code get ERP5 connection paramater from slappart0

• runner: API-fy inspectInstance

• promise.plugin: fix test ipv6_is_faster failing when ping outside is not possible

0.88 (2019-03-06)

• runner: add logging when we remove all instances inside a webrunner.

• runner: cleanup: do not use list as a variable

0.87 (2019-02-28)

• promise check_port_listening: port parameter should be int or str, unicode is not accepted

• monitor.runpromise: exit with non zero code if a promise has failed

0.86 (2019-02-06)

• promise: migrates old promises used in caddy-frontend SR to new format

0.85 (2019-01-09)

• promise: add new promise to check long request

• runner: display connection parameters sorted in alphabetical order

0.84 (2018-12-13)

• improve backupserver_check_backup promise

• fix resiliency tests

• add UTC timezone in promises logs (“+0000”)

0.83 (2018-11-14)

• Add scripts in python to be used in the resilience

• Usability improvement in the promess framework

• Fix flacky tests

0.82 (2018-10-26)

• 2 new promises: check_port_listening and backupserver_check_backup

• runner: add favicon

• resiliencytest: several fixes

0.81 (2018-08-29)

• promise: fix bug in check_free_disk promise: use ‘==’ instead of ‘is’ to compare unicode string

0.80 (2018-08-29)

• promise: fix move check_free_disk_space promise to plugin promise

0.79 (2018-08-29)

• promise: move check_free_disk_space promise to plugin promise with improvements

• runner: fix creating software link folder which cause rebuild of sr when /isRSReady is called

0.78 (2018-08-13)

• promise: check_free_disk does not return error if no data is found in collect db

0.77 (2018-07-31)

• runner: always start slaproxy service in webrunner

• promise: add new promise to check cpu load based on a threshold value

0.76 (2018-04-09)

• promise.plugin: uses ‘extra_config_dict’ to send custom promise parameters

0.75 (2018-03-30)

• monitor: reimplement monitor following the new promise design in slapgrid

• new promises, to check memory, check monitor bootstrap and partition state

• fix ‘attempt to write a readonly database’ in some promises and monitor.collect

0.74 (2018-02-12)

• qemu qmpclient: add support for cpu and memory hotplug

• monitor: Add a random delay of maximum 60 seconds to all monitor cron jobs

• monitor: use checkpromise method from slapos.core package

0.73 (2017-10-06)

• make monitor.collect use slapos.collect.db API’s

• new promise to monitor memory

• promise: fix most issues in apachedex and slowquery

0.72 (2017-09-25)

• securedelete: add securedelete script which use shred to securely wipe files

0.71 (2017-09-12)

• promise: Add ptdigest and apachedex result check promises with tests

0.70 (2017-08-18)

• monitor: Handle empty files case before parse json.

• monitor: Prevent to run collect when another instance is running

• monitor: Set timeout when getting sub monitor title

• monitor: Fix the report content output

• monitor: Allow to view and modify empty monitor config parameter

• qemu backup: Check running job exit after a timeout

• apachedex: Initial commit of the wrapper

• agent: Handle not found Software Installation

• agent: test_mapping should containts all the tests on each groups

0.69 (2017-06-01)

• promise.check_error_on_apache_log: Fix partial line parsers.

• promise.apache_mpm_watchdog: Implement watchdog for apache

0.68 (2017-05-11)

• slapos-kill: exclude self

0.67 (2017-04-18)

• promise.is_process_older_than_dependency_set: Verbose a bit of information when fail.

• monitor: Bug fix _every_xx_hour

• monitor: Add missingok for log rotate, if he has no file to rotate.

• monitor: Automatic build statistics on promise hisotry for archive.

• monitor: Logrotate can call this script multiple times

0.66 (2017-04-02)

• promise: Include promises for test networks and apache log checks.

0.65 (2017-01-23)

• checkfeedaspromise: returns more information on failure

• monitor: run command with low priority using nice

0.64 (2017-01-31)

• monitor: merge status2rss with monitor state generation script in order to have consistent result

• generatefeed: remove useless items instead of crashing

0.63 (2017-01-09)

• monitor: Allow to change promise timeout from UI

0.62 (2016-12-31)

• monitor_config_write: CORS apache conf should be rendered from a slapos template

• runner: user’s git information can now contain utf-8 characters

• monitor: fix cleanup of monitor promises result folder

0.61 (2016-10-26)

• monitor: allow to specify a second promise folder

• pubhubsub: new argument allowing to run the executable several times in case of failure

0.60 (2016-10-05)

• monitor: write pid to file while processing

• monitor: run all promises in a single crontask with timeout of 12 seconds per promises.

0.59 (2016-09-16)

• runner: add tests

• Add scripts to generate RSS feeds and to create promises on RSS feeds

• notifier: new arguments to generate verobose logging

0.58 (2016-09-07)

• monitor: multiple fixes and improvements

• resilient: fix tests

• agent: fix console script

0.57 (2016-08-09)

• monitor: Included monitor code with unit tests.

0.56 (2016-07-25)

• slaprunner: remove create user at first launch, clone repository can be done by slaprunner at startup.

• Remove slapos.builder and slapos.monitor as it only contains unused and dead code.

0.55 (2016-03-01)

• Switch to mysqlclient, that is a maintained fork of MySQL-python.

• Add an option in is-process-older-than-dependency-set to kill the process if older than dependency set.

0.54 (2016-02-25)

• Added is-process-older-than-dependency-set script.

0.53 (2015-11-26)

• networkbench: Minor fixes

0.52 (2015-09-02)

• testagent: Fully reimplementation

• networkbench: Accept external configurations and split file logs

0.51 (2015-08-17)

• networkbench: Include ping6 and TTFB support

0.50 (2015-07-20)

• slaprunner: Allow to configure the window title through ‘instance-name’ parameter.

0.49 (2015-07-17)

• slaprunner: filter out useless file extensions in treeview

0.48 (2015-04-09)

• slaprunner: Support command exiting with non 0 status in minishell

0.47.3 (2015-02-17)

• resiliency tests: hide output of slapos node instance.

• equeue: move takeover detection in thread lock.

• add erp5 resiliency test suite.

• harcode slaprunner request to use dev branch for erp5.

• slaprunner resiliency test suite: compatibility with python 2.7.9.

• resiliency test suite: do takeover from web interface.

0.47.2 (2015-02-14)

• equeue: prevent to run import script if takeover-triggered file exists.

• Advertise development of new version.

0.47.1 (2015-02-05)

• equeue: unlock lockfile at boot.

0.47 (2015-02-05)

• runner: fix syntax when stopping all processes.

• equeue: realtime logging.

0.46.1 (2015-01-30)

• runner: enhance software/instance explanation to user.

• runner: do not remove whole instance root directory, but only instance home directories.

• slaprunner resilient test: update API about slapgridResult.

0.46.0 (2015-01-16)

• slaprunner: don’t shutdown supervisord.

• slaprunner: Simplify process management and make sure ‘process killer’ handler works.

• webrunner: add getConnectionParameter API.

• equeue: create a lockfile when running importer command.

• Equeue: do not register command in database if it was not successful.

0.45.3 (2015-01-12)

• slaprunner: Quick and dirty support for ace editor autocompletion

0.45.2 (2014-12-02)

• resiliencytest framework: adapt abstract test suite to work with a single clone.

• resiliency test suites: send log to erp5 master.

• KVM resiliency test suite: use test framework

0.45.1 (2014-11-24)

• Fix logrotate and introduce a delay for networkbench.

0.45.0 (2014-11-24)

• Resiliency tests: allow to use from inside of a classical erp5testnode.

0.44.0 (2014-11-22)

• Add new networkbench command.

0.43.0 (2014-11-13)

• Add new slapos-kill command, deprecating killpidfromfile

• Make pycurl egg an optional dependency

0.42.0 (2014-10-28)

• The egg provides check-web-page-http-cache-hit script to check if a web page is well cached by Squid or TrafficServer.

0.41.0 (2014-10-20)

• The egg provides is-local-tcp-port-opened script to check if an ip address + port are currently opened on the local machine by reading in /proc/net/tcp and /proc/net/tcp6.

0.40.4 (2014-10-14)

• The equeue server now supports additional parameters to the callbacks, separated by 0 in the JSON protocol.

• The pubsubnotifier accepts a ==transaction-id parameter. If missing, the current timestamp is used.

• The pubsubnotifier then provides the transaction-id value in the POST call to /notify.

0.40.3 (2014-10-13)

• Slaprunner: stop proxy when removing instances content. This fixes the “partition11 table does not exist” error.

0.40.2 (2014-09-25)

• Slaprunner: Various bug fix

• Slaprunner: Update supervisorctl interaction to work with latest slapos.core revision

0.40.1 (2014-09-02)

• Slaprunner: Fix a bad behaviour within slapos

0.40 (2014-09-01)

• Slaprunner: Supervisord runs slapos commands

• Slaprunner: fixes for git repositories management

0.39.5 (Unreleased)

• Slaprunner: Add slave support test.

0.39.4 (2014-08-11)

• Slaprunner: Add a minishell feature, using far less bandwith than shellinabox

• Slaprunner: Slapgrid state is correctly rendered while compiling

• Slaprunner: consider new parameter’s values without needing to restart

• Slaprunner: slapgrid’s last result is always accessible through a stored file

• Slaprunner: Fix dangerous bug concerning custom parameters

• Slaprunner: various fixes and improvements

0.39.3 (2014-07-11)

• Slaprunner: Add folder to favorites

• Slaprunner: Fix https git clone not possible if no login

• Slaprunner: Improve editor full window

• Pubsub: More explicit RSS displayed in reversed chronoligical order

• Pubsub-notifier: add logger

• Slaprunner: Various fixes

0.39.2 (2014-05-16)

• Slaprunner: Add tab support

• Slaprunner: Add upload file

• Slaprunner: Improve editor full window

• Slaprunner: Various fixes

0.39.1 (2014-04-30)

• Slaprunner: Various fixes

0.39 (2014-02-20)

• Slaprunner: new web interface design

• Slaprunner: one function handle both “run software” and “run instance”

• Slaprunner: building and deploying can be customized

• Slaprunner: adds a multi-user feature

• Slaprunner: add fullscreen mode for text edition

• Slaprunner: direct access to monitoring of running instance, if it exists

0.38.1 (2013-12-06)

• Slaprunner: do not delete proxy.db on each run software

0.38 (2013-12-03)

• Slaprunner: adds an integrated shell

• Slaprunner: uses basic authentification

• Slaprunner: adds automated deployment of a Software Release

• Slaprunner: flask development server replaced by Gunicorn, a WSGI server

• Slaprunner: new test scenario for auto-deployment

• Runner resiliencytestsuite: adds basic auth support

• Runner resiliencytestsuite: tests can be done on only one Slapos node

0.37.4 (2013-10-15)

• Improve QEMU QMP wrapper by adding drive-backup method and other helpers.

0.37.3 (2013-10-10)

• pubsub: don’t swallow output of subprocess to allow debug.

0.37.2 (2013-10-10)

• Add QEMU QMP wrapper.

• KVM resiliency test: update docstring about how to setup disk image.

• KVM resiliency test: change key for each clone.

0.37.1 (2013-10-03)

• pubsub notifier: handle timeout and other connection errors.

• equeue: cast str(timestamp) to please gdbm.

0.37 (2013-09-30)

• equeue: log output of subprocess.

• slaprunner: don’t send 200 when login is bad.

• Improve reliability of resiliency tests.

0.36 (2013-09-05)

• Add resiliency test system.

• Add kvm resiliency test suite.

• Add slaprunner resiliency test suite.

• slaprunner: don’t tell user false assumptions.

• slaprunner: allow profile/instance to be run by GET.

• slaprunner: remove boring username/password length check.

0.35.1 (2013-08-05)

• slaprunner: add support for reverse proxy.

0.35.0 (2013-07-08)

• Resiliency: equeue is rewriten extending SocketServer.ThreadingUnixStreamServer. So far outputs of commands invoked by equeue are redirected to /dev/null to avoid locking the whole process.

• Add htpasswd script. This script is meant to produce htpasswd file without installing apache.

• slaprunner: Grammar improvements.

0.34.0 (2013-03-28)

• runner: Log textarea is resizable.

• runner: Fix bug where slaproxy hangs: Redirect slapproxy output to stdout.

• lampconfigure: Add support for running SQL scripts in lampconfigure.

• runner: software_type now survives to slaprunner reboot.

0.33.1 (2013-02-12)

• runner: Add default values for git repository while cloning.

• runner: Don’t remove git repo if error occurred during clone.

• runner: Fix bug where software_type of main instance is impossible to set reliably.

0.33 (2013-01-24)

• agent: run tests in order.

• slaprunner: Add process killer killing all processes when instance is stopped.

• slaprunner: slapgrid-cp will be run at startup.

• slaprunner: Improved design.

• slaprunner: Don’t set arbitrary limitations on the complexity of password.

0.32 (2012-12-13)

• pubsub: support multiple notifications and callbacks.

• pubsub: print/return errors from subprocess or notifications.

• agent: ignore SR status, retrieve instance status with .getStatus().

• slapmonitor: added slapmonitor-xml function + entry point.

0.31.1 (2012-10-02)

• slaprunner: Add new file manager.

• slaprunner: Ask confirm before delete important folder.

0.31 (2012-10-02)

• slaprunner: Add correct support for software_type.

0.30 (2012-08-29)

• Add promise support on slapcontainer script.

0.29.1 (2012-08-23)

• Minor fix on slapcontainer.

0.29 (2012-08-22)

• Add slapcontainer stateless script.

0.28.2 (2012-08-17)

• Slaprunner: Move password, process pid and config files to etc_dir

0.28.1 (2012-08-15)

• Update ACE to 08.11.2012 and JQuery to 1.8.0.

0.28 (2012-08-09)

• slaprunner: add support for authentication management.

0.27 (2012-08-07)

• pubsub: add support for multiple notification servers.

0.26.1 (2012-07-09)

• runner: Add support to slapos.core 0.26.2

0.26 (2012-07-06)

• slaprunner now supports authentication and instance parameters.

• Redisign of Test Agent so that it supports proper instantiation test and report.

0.25 (2012-06-19)

• slapmonitor is fully functional.

0.24 (2012-04-19)

• Slap Test Agent : test software release installation and instance allocation

0.22 (2012-04-05)

• Install argparse if missing

0.21 (2012-04-04)

• zodbpack : simple tool to pack ZODB

0.20 (2012-03-28)

• slaprunner : fix recursive include of .gif files

0.19 (2012-03-28)

• slaprunner : Add Support for more instances at the same time.

• slaprunner : Add slapgrid process control and management.

• slaprunner : Add Cloud9 IDE.

• slaprunner : Add develop location of cloned git repository.

0.18 (2012-02-23)

• slaprunner : Add git commit and push support.

• slaprunner : Solve UI problems in project management.

• slaprunner : Solve bug where ACE modifies carrier returns.

• slaprunner : Add ajax view of build log.

• slaprunner : Allow to edit any file in repository.

0.17 (2012-02-10)

• slapbuilder: Update slapbuilder to new OpenSuse 12.1.

• slapbuilder: Added automatic IPv6 support thanks to ViFiB OpenVPN.

• slapbuilder: Added virtual machine support.

• slapbuilder: Added “One Disk” feature.

0.16 (2012-01-20)

• Fix MANIFEST.in

0.15 (2012-01-20)

• slaprunner: Adding project and git repository management

0.14 (2012-01-09)

• onetimeupload: SSL should be provided by an http frontend

0.13 (2011-12-30)

• builder: fix usage of mkstemp

0.12 (2011-12-29)

• builder: dynamic offset calculation

0.11 (2011-12-23)

• equeue: Add execution queue server.

• pubsub: Add pubsub server using pubsubhubbub for notifications.

0.10 (2011-11-07)

• slapbuilder: Support banging.

• lamp : Fix bug when using file_token option

0.9 (2011-10-29)

• Add LAMP tool to manage lamp instances.

• Add onetimedownload : a simple script which download any http resource and save it into a specified file_path.

0.8 (2011-10-24)

• Add new style to runner.

0.7 (2011-10-17)

• generic: Added systool with killpidfromfile.

• slaprunner: Avoid error when trying to remove instance directory containing not writable directories.

0.6 (2011-07-13)

• slapbuilder: Typo during PKI repository creation fixed.

0.5 (2011-07-12)

• slapbuilder: Do not play too much with buildout, SlapOS installation is responsible for correctly featured zc.buildout

0.4 (1011-07-11)

• Update slaprunner to run on newer slapos (requires at least 0.10 to work).

0.3 (2011-06-27)

• slapbuilder: use slapos provided networkcached buildout.

• slapbuilder: fix network issues on busy networks (“Neighbour table overflow.”)

0.2 (2011-06-10)

• In slapbuilder: drop to 100 partitions by default, move out PKI from USB key to real disk.

0.1 (2011-05-27)

• Created slapos.toolbox from slapos.tool.builder, slapos.tool.runner, slapos.tool.cloudmgr, slapos.tool.ontimeupload and slapos.tool.networkcache.