Products.TextIndexNG3 3.4.8

What is TextIndexNG V3?

TXNG 3 is the reimplementation of the well-known TextIndexNG product for Zope 2 using Zope 3 technologies. The current implementation runs out-of-the-box on Zope 2 (in combination with Five). The core implementation can be re-used easily in Zope 3.

Features

• multiple-field indexing (e.g. you can create one index to index content by title, author and body and perform queries against each field)

• multi-lingual support (Products.LinguaPlone and plone.app.multilingual)

• pluggable components (storages, lexicons, query parsers, splitters, stopwords, normalizers)

• complex queries (AND, OR, NOT, phrase search, right truncation, wildcards, similarity search)

• indexes foreign formats (DOC, PDF, XML, SGML, PPT etc.)

• optional query autoexpansion to improve search results

Requirements

• Zope 2.10+

• for running TextIndexNG3 with Plone, you need Plone 3.1 or higher

Download and project area

TextIndexNG 3 is currently hosted on Github:

https://github.com/zopyx/Products.TextIndexNG3

Support

Bugs, support issues are handled for free soley either through the TextIndexNG 3 bugtracker on Github:

https://github.com/zopyx/Products.TextIndexNG3/issues

Dedicated commercial support is available on a per-hour or per-issue basis from http://www.zopyx.com/.

The latest source code is available from Github

https://github.com/zopyx/Products.TextIndexNG3

License

TextIndexNG 3 is published under the Zope Public License V 2.1 (see ZPL.txt) Other license agreements can be made. Contact us for details (info@zopyx.com).

TextIndexNG 3 contains copies of ZCTextIndex/WidCopy.py and ZCTextIndex/NBest.py which are published under the Zope Public License ZPL.

TextIndexNG 3 ships with a copy of the Snowball code (snowball.tartarus.org) for implementing stemming. This code is (C) 2001, Dr. Martin Porter and published under the BSD license.

TextIndexNG3 ships with a modified version of PLY, (C) 2001, David M. Beazley. PLY is published under the GNU Lesser Public License (LGPL).

TextIndexNG3 ships with the python-levenshtein extension written by David Necas und published under the GNU Public License (GPL).

Credits

Many thanks to Yvo Schubbe for contributing a lot of code for the 3.2.0 release. Thanks to Christian Zagrodnik for working on the 3.4.0 release.

Contributions

Third-party contributions that become part of the TextIndexNG3 core (means they are being checked into the TextIndexNG3 source repository) must be made under the same license as the source (ZPL 2.1). Contributed code is subject to be relicensed without further notice of the orginal contributor.

Installation

External converters

To convert foreign formats like .doc, .pdf etc. you need to install some external converters. See

http://zopyx.com/projects/TextIndexNG3/documentation

for details.

How to make your custom content-types searchable

Most current Zope index implementations are built on the fact that an index with id XX tries to lookup the indexable content either from an objects XX attribute or by calling the method XX() of the object. Although TextIndexNG V3 still supports this behaviour, the recommended way to make custom types indexable through TXNG3 is through providing dedicated methods that return indexable content. The API of these methods is defined in src/textindexng/interfaces/indexable.py. Custom types must either implement the IIndexableContent API directly or provide the interface through an adapter registered through ZCML. The IndexContentCollector class should be used to return indexable content either as unicode string or as binary stream (to be transformed through external converters). Some example how to use the indexing API can be found in src/textindexng/tests/mock.py (see classes Mock, MockPDF and StupidMockAdapter)

How to query the index?

TextIndexNG accepts multiple query options that influence the search results (options passed to the search() method):

• query - the search query (see below). Warning: the search query must always be a Python unicode string

• parser - id of a registered parser (default: txng.parsers.english)

• language - one of the languages registered for a particular index (default: the first registered language)

• field - perform query against this field (as registered with the index)

• similarity_ratio - a float value between 0.0 and 1.0 to determine the ratio for measuring the similarity of terms based on the Levenshtein distance (default: 0.75)

• autoexpand - ‘off’|’always’|’on_miss’ (default: ‘off’) determines how query terms are treated. ‘on_miss’ expands the query terms to all terms that are similiar to the original search term (if could not be found). ‘always’ expands the terms always. ‘off’ turns off auto-expansion. Auto-expansion helps you to improve the search result e.g. for mis-spelled words. Using auto-expansion might slow down the query performance.

• ranking - 0|1|True|False enables/disables (default 0|False) relevance ranking based on the cosine measure. Using ‘ranking’ requires that the index uses storage(s) that implement IStorageWithTermFrequency (txng.storages.term_frequencies). By default an index does not use this storage (see ‘storage’ parameter of the index constructor).

• ranking_maxhits - the maximum number of documents obtained from the ranking (default 50). ‘ranking’ must be set to True to use this option otherwise an exception will be raised.

Parsers

TextIndexNG comes with five query parsers. Each of them implements a different query syntax:

• txng.parsers.en - implements the query syntax as described below (this is the default query parser)

• txng.parsers.de - same as txng.parsers.en but it uses ‘UND’, ‘ODER’ and ‘NICHT’ instead of ‘AND’, ‘OR’ and ‘NOT’ (german parser)

• txng.parsers.fr - same as txng.parsers.en but it uses ‘ET’, ‘OUT’ and ‘PAS’ instead of ‘AND’, ‘OR’ and ‘NOT’ (french parser)

• txng.parser.dumb_and - this is a very simple parser that accepts only a whitespace separated list of terms which are all combined using AND. No fancy query options as with the parsers above are allowed.

• txng.parser.dumb_or - this is a very simple parser that accepts only a whitespace separated list of terms which are combined using OR. No fancy query options as with the parsers above are allowed.

Stemming

Wikipedia defines:

A stemmer is a program or algorithm which determines the
morphological root of a given inflected (or, sometimes, derived)
word form -- generally a written word form.

TextIndexNG V3 includes the Snowball stemmer library written by Martin Porter (snowball.tartarus.org) which provides stemming support for eleven languages. Stemming is an optional feature and must be specified when you create a new index. But note that stemming is incompatible by design with a number of TextIndexNG’s features. In general all features except searching for words without wildcards, left/right truncation won’t work and raise an exception.

Thesaurus

A thesaurus maps a query term to a sequence of related terms that will be used for searching. Therefore a thesaurus only affects searching but not indexing. Thesaurus are configured as named utitities implementing IThesaurus. The name of a configured thesaurus is by convention ‘txng.thesaurus.XXX’ where XXX is a country code. Multiple thesauruses for one language can be configured under different names. To tell TextIndexNG3 to use a thesaurus while searching you must use the name of a thesaurus or a list of thesaurus names as ‘thesarus’ parameter to the search() method.

Example:

index.search(some_query, thesaurus=('txng.thesaurus.de', 'txng.thesaurus.de-special'), ..)

In this case TextIndexNG will use all query terms from 'some_query' and all related
terms from a lookup in the configured thesauruses *.de and *.de-special.

Limitation:

Using a thesaurus is not compatible with phrases searches. Terms appearing within a phrase search will never be used for a thesaurus lookup.

Running the unittests:

bin/zopectl test -s Products.TextIndexNG3

Contact

CHANGES

3.4.8 (2014/04/21)

• fixed unit test failing between Plone 4.3.2 and Plone 4.3.3

3.4.7 (2013/05/26)

• workaround for plone.app.multilingual misbehaviour in cmf_adapters

3.4.6 (2013/05/26)

• fixed deprecation warning about getSiteEncoding()

3.4.4 (2011/06/15)

• fixed version number in metadata.xml

3.4.4 (2011/06/15)

• fixed version number in metadata.xml

3.4.3 (2011/05/31)

• fixed documenation issues

3.4.2 (2011/05/31)

• fixed CMFContentAdapter fallbacks added in release 3.4.1

• improved test setup

3.4.1 (2010/10/30)

• fixes in cmf_adapters.py for plone.indexer

• some Python 2.6 compatiblity fixes

• fixed some deprecation warnings

3.4.0 (2010/07/13)

• official release (same as 3.4.0)

3.4.0b1 (2010/05/20)

• Made the ranking method configurable on add.

• Allow to specify search_all_fields in query.