erp5.util 0.4.53

erp5.util
=========

Package containing various ERP5 related utilities.

Modules documentation
=====================


erp5.util.taskdistribution
--------------------------

Module to access TaskDistributor, used to run test on several machines
and aggregating results.
Use pydoc to get module documentation and usage example.


API Documentation
-----------------

You can generate the API documentation using `epydoc`::

$ epydoc src/erp5


testnode
--------


Utility able to call wget and varnishlog to extract Headers and return all failures
according expected caching policy.

This utility is configurable through a configuration file like::

[web_checker]
url = http://www.example.com/
working_directory = /home/me/tmp/crawled_content
varnishlog_binary_path = varnishlog
email_address = me@example.com
smtp_host = localhost
debug_level = debug

[header_list]
Last-Modified = True
Cache-Control = max-age=300
max-age=3600
Vary = Accept-Language, Cookie, Accept-Encoding
Accept-Language, Cookie
Accept-Language,Cookie,Accept-Encoding
Accept-Language,Cookie
Expires = True

[header url=.*/sitemap]
Last-Modified = True

[header content-type=.*/javascript]
Last-Modified = True
Cache-Control = max-age=3600
Expires = True

[no_header content-type=(image/.*|text/css)]
Vary = None

[erp5_extension_list]
prohibited_file_name_list = WebSection_viewAsWeb
Base_viewHistory
list
prohibited_folder_name_list = web_page_module
document_module


with::

url : website to check
working_directory : fetched data will be downloaded
varnishlog_binary_path : path to varnishlog
email_address : email address to send result
smtp_host : smtp host to use
debug_level : log level of this utility (debug =>very verbose,
info=>normal,
warning=>nothing)

header_list : Key == Header id.
value: if equals to True, it means that header needs to be present in RESPONSE
if it is a tuple, the Header value must sastify at least one of the proposed values

erp5_extension_list: Optional section.
prohibited_file_name_list: which check that any links redirect to prohibited forms
like WebSection_viewAsWeb, Base_viewHistory, list, ...
prohibited_folder_name_list: usefull to detect that links does not redirect to
specified modules like, web_page_module, document_module, ...


This utility requires wget => 1.12
And a callable varnishlog.
The utility must be run on same server where varnish is running.

web_checker reads varnishlogs to detect if a Query goes to the backend.

Changes
=======

0.4.53 (2018-09-07)
-------------------

* erp5.util

- testnode: give project title to runTestSuite [Sebastien Robin]
- testnode: support chmod'ed files during directories cleanups [Jerome Perrin]

0.4.52 (2018-08-21)
-------------------

* erp5.util

- Make scalability testing framework more stable. Stop using a dummy frontend master
and use host.vifib.net frontend with a valid SSL certificate instead. Always use
https.
[Yusei Tahara]

0.4.51 (2017-07-17)
-------------------

* erp5.util

- scalability testing framework [Roque Porchetto]

0.4.50 (2017-11-22)
-------------------

* erp5.util.testnode

- call only methods on Distributor [Lukasz Nowak]

0.4.49 (2017-05-11)
-------------------

* erp5.util.taskdistribution:

- Wrap in xmlrpclib.Binary if needed

0.4.48 (2017-04-20)
-------------------

* erp5.util.testnode:

[Yusei Tahara]
[Yusei Tahara]
- fix values of --firefox_bin and --xvfb_bin [Julien Muchembled]

0.4.47 (2017-04-05)
-------------------

* erp5.util.testnode:

- Make it more robust in cases where we have from time to time failures [Sebastien Robin]
- cosmetic: avoid -repository suffix [Julien Muchembled]

0.4.46 (2016-09-29)
-------------------

* erp5.util.testnode:

- Include js-logtail at the MANIFEST.in

0.4.45 (2016-08-05)
-------------------

* erp5.util.testnode:

- Do not block all test suites if one of them define broken repository [Sebastien Robin]
- Make sure proxy is really dead before starting new one [Sebastien Robin]

0.4.44 (2016-03-22)
-------------------

* erp5.util.testnode:

- Cancel test result if testnodes are unable to create partitions and unable
to find runTestSuite command.
- Set specific environment variable to build NumPy/friends & Ruby gems in
parallel.
- For local repositories, ignore revision defined in software release.
- Make it possible to define slapos parameters in test suites.

0.4.43 (2015-09-02)
-------------------

* erp5.util

- Make services much more reactive when server is back [Sebastien Robin]

* erp5.util.testnode
- Simple log viewer app not to download the whole suite.log [Jérôme Perrin]
- Make code more robust when checkout git files [Sebastien Robin]

0.4.42 (2014-12-02)
-------------------

* erp5.util.testnode

- Typo [Jérôme Perrin]
- Run first found runTestSuite from lowest matching partition, not random one [Cédric de Saint Martin]

* erp5.util

- Drop support for Python < 2.7 [Julien Muchembled]

0.4.41 (2014-08-07)
-------------------

* erp5.util.testnode

- Fix running test location [Rafael Monnerat]

* erp5.util

- Move dealShebang into Utils [Rafael Monnerat]

0.4.40 (2014-07-30)
-------------------

* erp5.util.testnode

- Bugfix for erp5/util/testnode/__init__.py [Rafael Monnerat]

0.4.39 (2014-07-30)
-------------------

* erp5.util.testnode

- update SlapOSControler cmd calls [Rafael Monnerat]

0.4.38 (2014-04-16)
-------------------

* erp5.util.testnode:

- cleanup after the merge of scalability code [Cedric de Saint Martin]

0.4.37 (2014-01-21)
-------------------

* erp5.util.scalability:

- New module [Benjamin Blanc]

* erp5.util.testnode:

- Minimize writes to storage holding MySQL databases.

0.4.36 (2013-06-30)
-------------------

* erp5.util.testsuite:

- delete git repos if url has changed [Sebastien Robin]

0.4.35 (2013-06-21)
-------------------

* erp5.util.testsuite:

- Fix additional_bt5_repository_id into testnode.py
[Benjamin Blanc]

0.4.34 (2013-04-11)
-------------------

* erp5.util.testsuite:

- allow to define sub results in tests, like we do for selenium
[Sebastien Robin]

0.4.33 (2013-03-14)
-------------------

* erp5.util.zodbanalyze:

- Initial version of an improved version of ZODB's ZODB/scripts/analyze.py
[Kazuhiko Shiozaki]

0.4.32 (2013-03-13)
-------------------

* erp5.util.testnode:

- add handling of httplib.ResponseNotReady error message [Sebastien Robin]
- do not fail when a different test suite repository branch is specified
[Sebastien Robin]

0.4.31 (2013-03-01)
-------------------

* erp5.util.testnode:

- after resetting software, retry_software_count was not resetted correctly
[Sebastien Robin]

0.4.30 (2013-02-20)
-------------------

* erp5.util.testnode:

- keep almost no tmp files, sometimes there is many Gb in /tmp after
one day [Sebastien Robin]

0.4.29 (2013-02-20)
-------------------

* erp5.util.testnode:

- make it able to resist to problems with slapos proxy when building
software [Sebastien Robin]

0.4.28 (2013-02-19)
-------------------

* erp5.util.testnode:

- make it able to resist to problems with slapos proxy [Sebastien Robin]

0.4.27 (2013-02-15)
-------------------

* erp5.util.testnode:

- testnode was still sometimes logging at several files at a time
[Sebastien Robin]

0.4.26 (2013-02-14)
-------------------

* erp5.util.testnode:

- do not reraise OSError when cleaning temp files

0.4.25 (2013-02-11)
-------------------

* erp5.util.testnode:

- close all timers when quitting, this makes stopping an erp5tetsnode
much faster [Sebastien Robin]
- remove hack on slapos/testnode after fix of slapos.cookbook [Sebastien Robin]
- remove old tmp files left by buildout (buildout has te bo fixed too)
[Sebastien Robin]
- remove logging handlers where the are not needed any more [Sebastien Robin]
- fixed the kill command, it was not able to kill properly childs [Sebastien Robin]

0.4.24 (2013-02-11)
-------------------

* erp5.util.testnode:

- Fixed wrong location for the construction os test suite software
[Sebastien Robin]

0.4.23 (2013-02-11)
-------------------

* erp5.util.testnode:

- Make erp5testnode allow remote access to test suite logs instead of
uploading them to master [Tatuya Kamada], [Sebastien Robin]


0.4.22 (2013-01-08)
-------------------

* erp5.util.taskdistribution:

- fix regression when used on Python < 2.7

0.4.21 (2013-01-07)
-------------------

* erp5.util.taskdistribution:

- really fix lock to avoid errors with concurrent RPC calls

* erp5.util.testnode:

- do not run test suites on deleted branches

0.4.20 (2012-12-19)
-------------------

* erp5.util.testnode:

- Make sure to kill grandchilds when killing a process [Sebastien Robin]

0.4.19 (2012-12-17)
-------------------

* erp5.util.testnode:

- Fixed undefined variable [Sebastien Robin]

0.4.18 (2012-12-14)
-------------------

* erp5.util.testnode:

- Solve ascii issues when deleting software [Sebastien Robin]

0.4.17 (2012-12-10)
-------------------

* erp5.util.testnode:

- Add thread Timer to terminate locked processes [Sebastien Robin]
- Add more unit tests [Pere Cortes]

0.4.16 (2012-11-14)
-------------------

* erp5.util.testnode:

- Improve handling of Xvfb and firefox [Sebastien Robin]
- check supported parameters of runTestSuite [Pere Cortes]
- add unit for runTestSuite [Pere Cortes]

0.4.15 (2012-11-07)
-------------------

* erp5.util.testnode:

- fixed profile generation when software repos is not defined first
[Sebastien Robin]
- ask wich test has priority to master more often [Sebastien Robin]

0.4.14 (2012-11-05)
-------------------

* erp5.util.testnode:

- force rebuilding software to avoid using old soft/code [Sebastien Robin]

* erp5.util.taskdistribution:

- handle another possible error with master [Sebastien Robin]

0.4.13 (2012-10-31)
-------------------

* erp5.util.testnode:

- Add unit test for erp5testnode (with some hardcoded path that
needs to be fixed ASAP) [Sebastien Robin]
- Split long functions into several more simple ones for code
simplicity and readability [Sebastien Robin]

0.4.12 (2012-10-25)
-------------------

* erp5.util.testnode:

- Fixed several issues introduced by the management of test
suite by the master [Sebastien Robin]

0.4.11 (2012-10-22)
-------------------

* erp5.util.testnode:

- Take test suite parameters from the master, to allow distribution
of the work by the master [Pere Cortes], [Sebastien Robin]

0.4.10 (2012-10-01)
-------------------

* erp5.util.testnode:

- Allow to use a firefox built by testnode for
functional tests [Gabriel Monnerat]

0.4.9 (2012-10-01)
------------------

* erp5.util.testnode:

- remove --now parameter when calling slapgrid-sr since
it is not yet well supported [Sebastien Robin]

0.4.8 (2012-09-27)
------------------

* erp5.util.testnode:

- use taskdistribution module to reduce code
[Vincent Pelletier], [Pere Cortes]

0.4.7 (2012-09-03)
------------------

* erp5.util.taskdistribution:

- work around test lines acquiring values from parent when no value is
provided. [Vincent Pelletier]
- fix a regression introduced in 0.4.6 which allowed parallel XMLRPC calls,
which is not supported. [Rafael Monnerat]

* erp5.util.benchmark:

- check whether at least one result file could be found when generating a
scalability report. [Arnaud Fontaine]
- make sure that diagram bars are properly aligned in scalability test
report. [Arnaud Fontaine]

* erp5.util.testsuite:

- new module [Rafael Monnerat]

0.4.6 (2012-08-10)
------------------

* erp5.util.taskdistribution:

- set socket timeout for RPC calls to prevent a deadlock happens.
[Rafael Monnerat]

0.4.5 (2012-07-04)
------------------

* erp5.util.taskdistribution:

- xmlrpclib does not support named parameters, use positional ones
[Vincent Pelletier]

0.4.4 (2012-07-04)
------------------

* erp5.util.taskdistribution:

- New module [Vincent Pelletier]

0.4.3 (2012-04-24)
------------------

* erp5.util.testnode:

- Improve detection of the cancellation of a test on the master
- better management of SIGTERM signal
- cleanup test instances to make sure nothing stay from a previous
test run

0.4.2 (2012-04-11)
------------------

* erp5.util.testnode:

- Improve testnode logs
- add a thread to upload ongoing logs to the master regularly
- if the software release is not built successfully after a
few time, totally erase software. This help unblocking if
buildout is unable to update software.
- check if the last test result was cancelled in order to
allow relaunching test without restarting testnode

0.4.1 (2012-02-29)
------------------

* erp5.util.testnode:

- Improve testnode's reliability when contacting remote master
- Try to build software releases multiple times before giving up


0.3 (2011-12-23)
----------------

* erp5.util.webchecker:

- Imported from https://svn.erp5.org/repos/public/erp5/trunk/utils/
Utility to check caching policy of websites

* erp5.util.testnode:

- improve logging [Sebastien Robin]
- fix passing bt5_path [Gabriel Monnerat]
- fix profile_path concatenation [Nicolas Delaby]
- fix git updating and parsing repository paths [Julien Muchembled]

* erp5.util.benchmark:

- new utility, work in progress [Arnaud Fontaine]

0.2 (2011-09-20)
----------------

* Imported from https://svn.erp5.org/repos/public/erp5/trunk/utils/

- erp5.util.test_browser:
Programmable browser for functional and performance tests for ERP5
- erp5.util.benchmark:
Performance benchmarks for ERP5 with erp5.utils.test_browser

0.1 (2011-08-08)
----------------

* erp5.util.testnode imported from recipe-like slapos.cookbook
[Łukasz Nowak]