roundup 1.6.0

I’m proud to release version 1.6 of Roundup which has been possible due to the help of several contributors. This release contains important security enhancements, so make sure to read docs/upgrading.txt to bring your tracker up to date. Other changes, as usual, include some new features and many bug fixes.

Features:

• issue2550894: migrate test suite and run_test.py to py.test (John Kristensen)

• issue2550880: Ability to choose password store scheme and SSHA support. Discussion on devel list is tending in favor of this patch. Embedded test works, my manual test with a SSHA password assigned to a user allowed the user to log in. Ran the test suite and the tests that were not skipped passed. (applied by John Rouillard)

• New Link/Multilink property attribute ‘msg_header_property’, can be used to configure additional headers in outgoing emails. See documentation in doc/customizing.txt. (Ralf Schlatterbeck)

• Allow multiple file uploads: If the html template specifies multiple=”multiple” for a file upload the user can attach multiple files and the form parser now handles this. (Ralf Schlatterbeck)

• issue2550886: Add support for an integer type to join the existing number type. This can be used for properties used for ordering, counts etc. where a decimal point isn’t needed. Developed by Anthony (antmail). Doc updates written by John Rouillard. (applied by John Rouillard)

• Updated html/_generic.404.html to use the page template. So 404 errors now include the left hand menu, a proper page title and body content. Note added to doc/upgrading.txt on how to add it to deployed trackers. (John Rouillard)

• issue2109308 - Allow subject of nosy messages be changed from reactor Adds a subject parameter to nosymessage function. Patch initally generated by Frank Niessink. Tests, adaptation by John Rouillard.

• issue2550683 Allow indexargs_form filter variable exclusion. Patch generated by Bruce Tulloch (bruce). Applied and docstring for indexargs_form updated by John Rouillard. Patch description is: This is required to allow indexargs_form to be used in conjunction with other form variables which replace some filterspec parameters.

  One must exclude all variables from the indexargs_form call which are to be replaced with values that are derived from other form input elements, otherwise they will clash with the “hidden” input elements generated by indexargs_form itself.

  For example:

  <tal:block replace="structure python:request.indexargs_form(
                    sort=0,group=0,filter=0,columns=0,
                    exclude=['type','status','assignedto'])"/>

  where the variables type, status and assignedto are supplied via other form input elements. Without the new exclude argument to indexargs_form, all hidden input elements otherwise generated by this call would need to be manually added to the template code. Further, given that the template may not know what other variables may be defined, it may not even be possible to code this without some python helpers. [rouilj I think this is an example usecase. Possible assignedto users need to have a specific role. Create TAL that filters the users to the select few. Defines a select list for assignedto. Use exclude=[‘assignedto’] to prevent the indexargs_form from generating a confliciting assignedto field which lists all users regardless of the role.]

• allow user to recover account password using an entry in the Alternate E-mail addresses list. See:: http://psf.upfronthosting.co.za/roundup/meta/issue564 for description. Merge request at:: https://sourceforge.net/p/roundup/code/merge-requests/1/ Patch supplied by kinggreedy. Applied/tested by John Rouillard

• issue2550636, issue2550909: Added support for Whoosh indexer. Also adds new config.ini setting called indexer to select indexer. See doc/upgrading.txt for details. Initial patch done by David Wolever. Patch modified, docs added and committed by John Rouillard.

• issue2550803: Replying to NOSY mail goes to the tracker through reply-to, not original message author. Created new [tracker] replyto_address config.ini option to allow: 1) setting reply-to header to the tracker 2) setting reply-to header to the address of the author of the change 3) setting it to a fixed address (like noreply@some.place) Done by John Rouillard from proposal by Peter Funk (pefu) in discussion with Tom Ekberg (tekberg). See doc/upgrading.txt.

• issue1714899: Feature Request: Optional Change Note. Added a new quiet=True/False option for all property types. When quiet=True changes to the property will not be displayed in the:

  * confirmation banner (shown in green) when a change is made
  * property change section of change note (nosy emails)
  * web history display for an item.

  Note that this may confuse users if used on a property that is meant to be changed by a user. It is most useful on administrative properties that are changed by an auditor as part of a user generated change. Original patch by Daniel Diniz (ajaksu2) discussed also at:

  http://psf.upfronthosting.co.za/roundup/meta/issue249

  Support for setting quiet when calling the class specifiers. E.G. prop=String(quiet=True) rather than:

  prop=String()
  prop.quiet=True

  support for anydb backend, added tests, doc updates, support for ignoring quiet setting using showall=True in call to history() function in templates by (John Rouillard). (Note implementation changed while implementing fix for issue2550864. Filtering of quiet properties pushed down to the hyperdb.py::Class::history function. This fixes a small bug in the implementation that caused a limiting the templating history call to display fewer than the requested number of items if some were quiet.)

• issue2550767: Add newitemcopy.py detector to notify users of new items. Added to detectors directory and a README.txt generated to describe the purpose of the directory. It also says the detectors are provided on an as-is basis and may not work. Detector by W. Trevor King (wking), rest by John Rouillard.

• issue934009: Have New Issues Submitted By Email Not Change Body! The mailgw config options: keep_quoted_text and leave_body_unchanged can now have a new values: new. If set to new, keep_quoted_text acts like yes if the message is starting a new issue. Otherise it strips quoted text. This allows somebody to start a new issue by forwarding a threaded email (with multiple quoted parts) into roundup and keeping all the quoted parts. If leave_body_unchanged is set to new, even the signature on the email that starts a new issue will be preserved.

• New cgi action restore (RestoreAction) which reverses the effects of the retire action. Created while implementing fix for issue2550831. Requires restore permission in the schema. See upgrading.txt for migrating to 1.6.0 for details. (John Rouillard)

• issue2550751: Email Header Issue. Noel Garces requested the ability to suppress email headers like “x-roundup-issue-files”. With Ralf’s addition of the Link/Multilink property attribute ‘msg_header_property’ we can do this easily. Setting the ‘msg_header_property’ to the empty string ‘’ (not to None) will suppress the header for that property. (John Rouillard)

• issue2550891: Allow subdir in template value. Anthony (antmail) requested the ability to put templates into subdirectories. So the issue class can accept @template=issues/item to get the html/issues/issue.item.html template. See doc/upgrading.txt.

• issue1842687: Keywords: After creating, stay in “Create New” mode. Change to classic tracker template to provide a check box (checked by default) that keeps the user on the “Add new keyword” page after submitting a new keyword. Usually after submission, you will see the page for the new keyword to allow you to change the name of the keyword. (John Rouillard)

• issue2550757 - internal restructuring to allow admin.py to be tested more easily. W. Trevor King (wking)/ John Rouillard.

• When storing user-defined queries we now store the template with the query if the template name is different from ‘index’. This allows stored queries for templates different from the default ‘index’ template. (Ralf Schlatterbeck)

• Number properties now have an optional attribute use_double to request double precision float as the storage type for this property. (Ralf Schlatterbeck)

• issue2550796: Calendar and Classhelp selection tools don’t cause onchange event to be triggered. Using the helper popups for modifying lists of users, lists of issues, dates etc.. now trigger the change event on the form’s field. This allows onchange javascript to trigger to highlight changes, recalculate other form values etc. See upgrading.txt for details on applying these changes to your tracker. (John Rouillard)

• menu template function has a new parameter “showdef”. When set to a string, the string is appended to the displayed option value. This allows the user to reset the value for the menu (select) to the original value. (John Rouillard)

• @template html url parameter can be set to “oktmpl|errortmpl”. When a form is submitted, if the form passes validation the oktmpl is used for the resulting page. If the form fails submission the errortmpl page is used to display the form. The errortmpl will usually be the same template used to edit the form. See the section on “Implementing Modal Editing Using @template” in customizing.txt. (John Rouillard)

• New form of check function is permitted in permission definitions. If the check function is defined as:

  check(db, userid, itemid, **ctx)

  the ctx variable will have:

  ctx['property'] the name of the property being checked or None
  ctx['classname'] the class that is being checked or None
  ctx['permission'] the name of the permission (e.g. View, Edit)

  At some future date the older 3 argument style check command will be deprecated. See upgrading.txt for details.

• New property for permissions added to simplify the model. See customizing.txt and search for props_only and set_props_only_default in the section ‘Adding a new Permission’. (John Rouillard)

• issue2550690 - Inadequate CSRF protection. Improvements in Cross Site Request Forgery protection to check HTTP headers and nonces. If the header/nonce is present, they are validated. But if headers or nonces are missing access is granted. The enforcement policy can be set in config.ini. Requiring enforcement will need some changes to templates. Support for protecting xmlrpc endpoint not well tested. See upgrading.txt. (John Rouillard)

• Added support for using the SameSite cookie option on the session cookie. Default is lax, but there is a settable option in config.ini file to change to strict or suppress it entirely. See upgrading.txt. (John Rouillard)

• Added a new roundup-admin command: updateconfig. Similar to genconfig but it uses values from an existing config.ini rather than default values. Use to update an existing config.ini with new options and help text. (John Rouillard)

• issue2550864: Potential information leakage via journal/history Hyperdb history function now only returns properties that the user can View or Edit and links to objects the user can see. Can be overridden by setting a parameter when calling the method. Also restructured code that implemented issue1714899 moving it from the templating class to the hyperdb. (John Rouillard)

• Improves diagnostics for mail processing: When using logging level = DEBUG, bounces and bounce problems are logged. (Bernhard Reiter)

• In roundup-server, pass X-Forwarded-For and X-Forwarded-Proto headers as the environment variables: HTTP_X-FORWARDED-FOR and HTTP_X_FORWARDED_PROTO. If the user is running roundup server behind a proxy, these headers allow the user to write extensions that can figure out the original client ip and protocol. None of the core roundup code uses these headers/env vars. These headers can be spoofed by bad proxies etc. so you have been warned.

• issue2550799: provide basic support for handling html only emails Emails missing text/plain parts but with text/html parts can be converted into text. If this is done the email will no longer be bounced back to the sender with an error. Enable by configuring the convert_htmltotext option in your upgraded config.ini. (Initial patch by Igor Ippolitov merged with changes by John Rouillard.)

• Add a ‘retired’ parameter to Class.filter to allow searching for retired, non-retired or all (retired and non-retired) items similar to the argument of the same name to Class.getnodeids. This is ‘False’ by default (finding only non-retired items for backwards compatibility) and can be set to None (for finding retired and non-retired items) or True (for finding only retired items).

• Requires Python 2.7 now, indicated in version_check.py and doc/installation.txt. (Bernhard Reiter)

• New -L flag to roundup-server to send http/https request logs through the python logger module (using roundup.http). This allows automatic log rotation. Without it, log file rotation requires restarting the server. (John Rouillard)

• Part of issue2550960. Applied patch 0038 to upgrade documentation code examples to support both python 2 and 3. (Joseph Myers)

Fixed:

• issue1615201: Optionally restore the original (version 0.6) mailgw behaviour of ignoring a Resent-From:-header and using the real From-header instead: new configuration option EMAIL_KEEP_REAL_FROM (Peter Funk aka Pefu).

• issue2550717: Changed a couple of residual email references into E-Mail in German translation (John Rouillard)

• issue2550669: Adding documentation for csv_field_size to the customizing tracker section of doc/customizing.txt (John Rouillard)

• issue2550601: gsoc-2009 “bug” class doesn’t have “patches” property Added multilink to patches to the bug schema in the devel template. (applied by John Rouillard)

• issue2550748: Crash when creating new issues with non-existing multilink values (in classic template). Applied patch so it now errors the same way as an update does. (applied by John Rouillard)

• issue2550757: one bug raised by issue fixed. Patch created by W. Trevor King (wking) for documentation of mailgw applied by John Rouillard.

• Fix processing of additional arguments to cgi method ‘menu’: This would not work if more than one additional argument is used. (Ralf Schlatterbeck)

• Update documentation of some existing property attributes (like ‘do_journal’ for Link/Multilink properties), this also adds missing documentation for issue1444214. (Ralf Schlatterbeck)

• issue2550763 Strip whitespace from Multilink values after + or -. (W. Trevor King) Test heavily modified by John Rouillard. (applied by John Rouillard)

• issue2550907 Fix errors when creating documentation. Work done by Peter Funk (pefu). (Applied by John Rouillard with small change omitting obsolete security.txt.)

• issue2550826 Capture some exceptions from auditors/reactors and raise a DetectorError instead. This allows failures like IOErrors from the detectors (e.g. unable to access files) to be handled. Previously an IOError just resulted in no output (premature end of headers under apache). Problem diagnosed and initial patch created by Tom Ekberg (tekberg). Further testing and patch change done by John Rouillard.

• issue2550851 in installation doc removed directions for installing additional codecs for Asian languages. They they appear to be part of the standard python since at least 2.6. Also the quoted url is obsolete. See ticket if you think you need the codecs.

• issue2550823 improve mailgw logging for node creation errors. Patch by r.david.murray (applied by John Rouillard).

• issue2550549 Postgres error on message templating Exception gets thrown and not captured if nodeid is too large on postgres. Added a check in rdbms_common layer that max nodeid is < 2^31 -1. Large nodeid now return no such id error upstream. Patch idea from: martin.v.loewis. (John Rouillard)

• issue2550723 Fix propagation of @pagesize When @pagesize=0 is specified (indicating show all), the value of pagesize is not propigated to the prev link. This patch fixes that. Patch provided by John Kristensen. (Applied, light testing by John Rouillard.)

• issue2550850 anypy/email_.py uses BSPACE which is not defined in python 2.7 Supplied a definition for BSPACE since it seems to not be defined anywhere. Reported by Dennis Boone. (John Rouillard)

• Validate properties specified for sorting and grouping in index views. Original patch from martin.v.loewis via: https://hg.python.org/tracker/roundup/rev/439bd3060df2 Applied by John Rouillard with some modification to properly identify if the bad property is a sort or grouping property. Tests added.

• Validate Integer and Numeric type filter parameters rather than passing output down to db level. Initial patch at: http://hg.python.org/tracker/roundup/rev/98508a47c126 by Martin.V.Loewis. Numeric test patch applied, Integer code and tests developed by John Rouillard.

• issue1926124: fix crash in roundup_admin migrate option. Patch submitted by Henry (henryl), modified value to False since this produces the correct “No migration action required” output from the migrate command.

• issue2161722: oudated docs (sic) Fix old entry in FAQ, update roundup-server config docs and example file from current roundup-server output. Update some typos in .py files. John Rouillard.

• issue2550572: setting nosy=+foo on multiple issues gives them all the same exact nosy list. Fixed a missing reinitialization that has to occur every time though the loop in do_set. Manual tests work. (John Rouillard)

• issue2550653: xapian search, stemming is not working This is a partial fix for the issue. It does make stemming work (so searching for silent will also return docs with silently in them). However to do this we need to lowercase the text so the porter stemmer will work. This means capitalization is not preserved. Fix done by David Wolever (wolever). Committed and doc updates John Rouillard.

• issue2550855: “show unassigned” link shows all open issues if not logged in. This adds permission for the anonymous user to search the users class. Without this the unassigned search can’t see if there is a user assigned to an issue, so it acts like all open issues. Patch supplied by Stuart McGraw (smcgraw). For caveats see upgrading.txt and the comments in the default templates. (Docs created and applcation by John Rouillard)

• issue2550854: including new field in All text* search. Fixed documentation in customizing.txt. The default for indexme on String fileds is ‘no’ not ‘yes’. So to get a new string field into the full text/all text index you need to use String(indexme=’yes’). Reported by Michael Belleville. (John Rouillard)

• issue2550853 - better error handling and cleanup on some postgres tests by Stuart McGraw.

• issue2086536 - back_postgresql: fixing pg_command and prefering psycopg2. Patch done by Philipp Gortan (mephinet). His patch also improves handling of retryable errors. Applied and edited by John Rouillard. Edits included removing support for psycopg1. See:

  https://sourceforge.net/p/roundup/mailman/message/32855027/

  for rational for dropping it.

• issue2550831: Make the classic template query.edit page work. Many fixes and improvements. See upgrading.txt for details. Diagnosis and fix with patch by R David Murray. Support for restoring retired but active queries, html layout changes and doc by John Rouillard.

• issue2550785: Using login from search (or logout) fails. When logging in from a search page or after a logout it fails with an error. These failures have been fixed. The fix also keeps the user on the same page they started from before the login. There are two parts to this: 1) changes to the templates to properly define the __came_from form element. See upgrading.txt. 2) code changes to the LoginAction code in roundup/cgi/actions.py. (John Rouillard)

• issue2550648 - partial fix for problem in this issue. Ezio Melotti reported that the expression editor allowed the user to generate an expression using retired values. To align the expression editor with the simple dropdown search item, retired values are now removed from the expression editor. (We have an open question as to whether this is desirable.)

• issue2550743 - Reindex with MySQL Server failed. It looks like indexing large documents may require increasing mysql’s max_allowed_packet setting. Documented the issue in doc/mysql.txt. Possible solutions include: increasing value of MySQL parameter, changing the full text search engine to whoosh or xapian. Problem report by telsch. Analysis/doc by John Rouillard.

• issue2550882. Reported by Karl-Philipp Richter. Fixed installation.txt documentation to include better directions on starting roundup-server on different ports/ip addresses. Also updated man page to include default use of localhost for -n and use of -n 0.0.0.0 to bind to all addresses on the host. (John Rouillard)

• issue2550827, issue2550718. Doc additions so people know that a python 32 bit installation may be required for windows. Additional documentation on the requirement of pywin32 for running roundup as a windows service. Also the windows installer must be run as administrator and strong encouragement for installing the pytz module added to doc/installation.txt.

• issue2550776: imapServer.py problem. Fixed a missing initialization of the logging level if no logging level option is supplied. (John Rouillard)

• issue2550839: Xapian, DatabaseLockError: Unable to get write lock on db/text-index: already locked. Put in a retry loop that will attempt to get the lock. Total delay approx 4.5 seconds. (John Rouillard)

• issue2550727: db.newid is broken with sqlite. Added proper transaction lock around the sql code to get a new id. The locking that pysqlite attempts had to be defeated because it is broken. Had to explicitly manage transactions with BEGIN IMMEDIATE and call sql_commit. Note that this reduces performance in return for accuracy. Problem reported by Matt Mackall (mpm) (John Rouillard).

• issue2550701: Path traversal from template names. This affects the tal based template engines (zopetal, chameleon). If a directory with a specific name is created in the html subdirectory, the template name in the url can be used to get access to files outside of the tracker html directory. This has been fixed by normalizing the path and comparing to the normalized path for the html directory. See doc/upgrading.txt. (John Rouillard)

• Fix subject parsing in mail gateway. The previous parsing routine would not ensure that arguments are at the end of the subject and when subject_suffix_parsing was configured to be ‘loose’ it would truncate the subject when encountering a double prefix, e.g. Subject: [frobulated] [frobulatedagain] this part would be lost (Ralf Schlatterbeck)

• issue2550795: @dispname query args in page.html search links not valid html. Some queries with names that include spaces are not properly url encoded/quoted. I.E. a space should be replaced with %20. Fixes to allow a url_query method to be applied to HTMLStringProperty to properly quote string values passed as part of a url.

• issue2550755: exceptions.NotFound(msg) msg is not reported to user in cgi. When an invalid column is specified return error code 400 rather than 404. Make error code 400 also return an error message to the user. Reported by: Bernhard Reiter, analysis, fix by John Rouillard.

• issue1408570: Finally fix that form values are lost on edit exceptions. This occured for example if editing an issue with the classic template and setting ‘superseder’ to a non-existing issue number. All changes to the form where the original field was non-empty were lost. (Ralf Schlatterbeck)

• Fix submit_once Javascript function: This needs to return a boolean value (not and integer like 0 or 1). And the work-around for an ancient version of Internet Explorer would make it break for a recent Firefox. The old version would show the popup but after clicking away the alert it would load the page. The new version (tested with Chromium and Firefox) doesn’t load the page. (Ralf Schlatterbeck)

• Fix Traceback in backends/portalocker.py on windows due to missing windll import, thanks to Heiko Stegmann for suggesting a first fix. (Ralf Schlatterbeck)

• issue2550933 - Fix Traceback in cgi/templating.py when a string is passed to PasswordHTMLProperty::plain. (John Rouillard)

• issue2550934 - templating.py-indexargs_form() returns id’s as space separated list not comma separated. This fixes the format of the id url parameter when generated by indexargs_form. (John Rouillard)

• issue2550932 - html_calendar produces templating errors for bad date strings. Fixed to ignore bad date and highlight todays date in the calendar popup.

• Query handling requires that query names for a user are unique. Different users are allowed to use the same query name. Under some circumstances a user could generate a second query with the same name. The SearchAction function has been corrected to report this error. Also the index.search.html template in the classic tracker and corresponding templates in the other example trackers has been modified to include:

  <input type="hidden" name="@template" value="index|search"/>

  so an error from SearchAction will display an error message and keep the user on the search page so they can correct the error. See doc/upgrading.txt. (John Rouillard)

• When a new named search is created, the index page that is displayed doesn’t show the name. This has been fixed by setting the @dispname to the query’s name. (John Rouillard)

• Passing args into indexargs_url(..,{‘@queryname’: request/dispname or None, ‘Title’: ‘some’ }) where the value of the arg is None will not add the arg to the url. In the example above @queryname will only be in the url if dispname is set in the request. (John Rouillard)

• The HTMLClass::properties() method produced a list of properties that the user could not search. As a result these properties can not be used for sorting or grouping index pages. This patch eliminates the confusion that results from this mismatch by verifying that all properties returned are searchable. (John Rouillard)

• Mutilinks can be displayed with their labelprop using the plain() method, but they can not be looped over using tal:repeat if the user doesn’t have view access to the class the multilink represents. The permissions check was changed to require that the user have View access to the labelprop for the class rather than View access to the class. (John Rouillard)

• issue2550937: fix crash by verifying that sendto is not null before calling mailer.smtp_send. Discovered and patched by Trent Gamblin. Applied by John Rouillard.

• removed old code from roundup-admin that implemented the obsolete config (do_config) command. (John Rouillard)

• Modified configuration option static_files to be a space separated list of directories to search for static files in the web interface. If one of the elements is -, the search stops and the TEMPLATES directory is not searched. See:

  https://sourceforge.net/p/roundup/mailman/message/35773357/

  subject is “showing template sources to all”.

• issue2550945: OpenPGP: Extends newissuecopy.py to encrypt if configured. (Bernhard Reiter)

• CSRF protection broke the retire function for query edit. Fix javascript and make sure csrf tokens are provided in the right places. (John Rouillard)

• query.item.html was missing checks to verify that a query should be visible to the user. This is fixed and users can only view queries that they own or that are not private. (John Rouillard)

• issue2550953: Patch: fix for context.is_view_ok check in jinja2 template Form controls are displayed when anonymous views indexes but is denied access. (patch by Anton Schur applied by John Rouillard)

• issue2550957: Duplicate emails (with patch). Bcc and cc users passed to nosymessage are not properly recorded. This results in duplicate emails. (patch by Trent Gamblin (trentgg) applied by John Rouillard).

• issue2550954: History display breaks on removed properties Now changes to removed properties, and link/unlink events from non-existing properties or classes no longer trigger a traceback. Concerning the visibility: We have a new config-item obsolete_history_roles in the main section that defines which roles may see removed properties. By default only role Admin is allowed to see these.

• Fix issue2550955: Roundup commits although a Reject exception is raised Fix the problem that changes are committed to the database (due to commits to otk handling) even when a Reject exception occurs. The fix implements separate database connections for otk/session handling and normal database operation.

• Allow empty content property for file and message via xmlrpc interface. This used to raise a traceback in the (sql) backend.

• Work around a limitation in python2.7 implementation of poplib (for the pop3 protocol for fetching emails): It seems poplib applies a line-length limit not just to the lines involving the pop3 protocol but to any email content, too. This sometimes leads to tracebacks whenever an email exceeding this limit is encountered. We “fix” this by monkey-patching poplib with a larger line-limit. Thanks to Heiko Stegmann for discovering this.

If you’re upgrading from an older version of Roundup you must follow the “Software Upgrade” guidelines given in the maintenance documentation.

Roundup requires python 2.7 or later (but not 3+) for correct operation.

To give Roundup a try, just download (see below), unpack and run:

python demo.py

Release info and download page:

http://pypi.org/project/roundup

Source and documentation is available at the website:

http://roundup-tracker.org/

Mailing lists - the place to ask questions:

http://sourceforge.net/mail/?group_id=31577