Solr integration for Plone
Project description
Introduction
ftw.solr integrates the Solr search engine with Plone.
ftw.solr does not try to replace the portal catalog of Plone and does not hook into the catalog’s search function. Instead it provides a search utiltity that must be used explicitly. It’s meant to be used in search forms for fulltext searches while the portal catalog is still in use for things like navigation or folder contents. The goal is to get rid off all fulltext indexes (e.g. ZCTextIndex) in the portal catalog.
ftw.solr requires Apache Solr 7.0 or higher.
Installation
Add as dependency
Install ftw.solr by adding it to the list of eggs in your buildout or by adding it as a dependency of your policy package.
[instance]
eggs +=
ftw.solrSolr installation
To install a Solr server with buildout you can use the ftw.recipe.solr recipe.
[solr]
recipe = ftw.recipe.solr
cores = mycoreConfigure the Solr connection
The connections settings for Solr can be configured in ZCML and thus in buildout. Example:
[instance]
zcml-additional =
<configure xmlns:solr="http://namespaces.plone.org/solr">
<solr:connection host="localhost" port="8983" base="/solr/mycore"/>
</configure>
By default, ftw.solr will do full text extraction by passing the blob’s filesystem path to the Solr Cell extract handler, assuming that Solr runs on the same machine and has access to the blob storage.
For setups where this isn’t desired, the connection option upload_blobs can be set to true in order to make ftw.solr upload the blobs directly to the extract handler via HTTP POST:
[instance]
zcml-additional =
<configure xmlns:solr="http://namespaces.plone.org/solr">
<solr:connection host="localhost" port="8983" base="/solr/mycore" upload_blobs="true"/>
</configure>
Run buildout
After running buildout and restarting your instance you can install the ftw.solr addon in Plone.
Python 3.8 - Plone 5.2
Install and run tests locally:
python3 -m venv .
./bin/pip install -r requirements.txt
./bin/buildout -c test-plone-5.2.x-py38.cfg
./bin/testUsage
Get the ISolrSearch utility and call the search method to get search results from Solr.
from ftw.solr.interfaces import ISolrSearch
from zope.component import getUtility
solr = getUtility(ISolrSearch)
resp = solr.search(query=u'SearchableText:foo')You can get a plone.app.contentlisting style result by adapting IContentListing:
from plone.app.contentlisting.interfaces import IContentListing
listing = IContentListing(resp)Solr Index Maintenance
For indexing Plone content and other maintenance work you can use the solr Zope command. Run bin/instance solr -h for available options.
Clear the Solr index:
bin/instance solr clearRebuild the complete Solr index:
bin/instance solr reindexReindex specific indexes:
bin/instance solr reindex -i modified createdSynchronize the Solr index with the portal catalog:
bin/instance solr syncLinks
Continuous integration: https://jenkins.4teamwork.ch/search?q=ftw.solr
Copyright
This package is copyright by 4teamwork.
ftw.solr is licensed under GNU General Public License, version 2.
Changelog
2.13.3 (2024-10-14)
Fix becoming the system user in zopectl command in Docker deployments. [buchi]
Fix finding Plone sites in zopectl command in Docker deployments. [buchi]
2.13.2 (2024-04-04)
Fix argument parsing of zopectl command if run with Zope2.Startup.zopectl. [buchi]
2.13.1 (2023-06-20)
Log warning when a search is performed without specifying fl. [njohner]
Fix solr sync maintenance command not committing all batches. [lgraf]
2.13.0 (2023-04-11)
Add support for Plone 5.2 and Python 3. [buchi]
Add Python 3.8 and up compatibility. [mathias.leimgruber]
2.12.1 (2022-08-03)
Make updates_in_post_commit_enabled() helper more robust. [lgraf]
2.12.0 (2022-07-19)
Add option to also check allowedRolesAndUsers when determining items not up to date in Solr. [njohner]
Add option to execute update commands in post commit hook. [njohner]
2.11.0 (2022-07-04)
Add support for language specific fields. [buchi]
2.10.0 (2021-09-21)
Add flag to force clearing solr even when solr is not enabled. [njohner]
2.9.5 (2021-08-17)
Add unrestricted_search method. [njohner]
2.9.4 (2021-07-29)
SolrDocument returns None for existing fields with missing value. [njohner]
2.9.3 (2021-04-22)
Fix indexing of SearchableText only for objects with primary field. [njohner]
2.9.2 (2021-01-13)
Do not ignore None values when indexing data in solr. [njohner]
2.9.1 (2020-12-15)
Cleanup extract_commands before sending them to solr. [njohner]
2.9.0 (2020-08-04)
Treat docs with no created field as out of sync in diff. [deiferni]
2.8.6 (2020-06-02)
Log maintenance commands to a dedicated logfile. [lgraf]
Add option to limit items logged in diff, switch default to only diff 5 items. [deiferni]
Add helper for creating Solr filters from catalog queries. [buchi]
2.8.5 (2020-05-06)
Make sure maintenance scripts are run as system user. [njohner]
2.8.4 (2020-04-02)
Fix update of the modified index. [njohner]
2.8.3 (2020-04-02)
Update the modified index to match new rounding of DateTime. [njohner]
2.8.2 (2020-03-27)
Fix to_iso8601 prevent rounding errors with DateTime milliseconds above 999500. [deiferni]
Fix blob extraction for DX items. [mbaechtold]
2.8.1 (2020-02-05)
Fix to_iso8601 for years before 1900. [deiferni]
2.8.0 (2019-12-06)
Add support for uploading blobs to extract handler. [lgraf]
Add support for Plone 5.0 and 5.1. [buchi]
2.7.0 (2019-09-16)
Add console script for maintenance tasks. [buchi]
Add registry setting for enabling/disabling indexing. [buchi]
2.6.2 (2019-09-12)
Avoid sending atomic update commands with null-documents. [lgraf]
2.6.1 (2019-08-26)
Fix ‘path_depth’ indexing handling, when ‘path_depth’ is not part of the schema. [phgross]
2.6.0 (2019-08-19)
Use Solr Cell only for text extraction. The SearchableText is now also indexed using the update handler which allows atomic updates. Also fixes #137. [buchi]
Fix error when trying to delete an object without a unique key. [jone]
2.5.0 (2019-07-02)
Make sure to always update ‘path_depth’ if ‘path’ is (re)indexed. [lgraf]
Handle facet_counts in solr response. [njohner]
2.4.0 (2019-06-13)
Perform soft commits and wait for new searcher by default to allow near real time (NRT) searching. [buchi]
Additionaly index metadata of files using an update command to make them visible as soon as possible. [buchi]
Flush optimize command. [buchi]
Add support for indexing path depth. [buchi]
2.3.2 (2019-04-29)
Fix partial reindexing (atomic updates) for fields without a value. [buchi]
2.3.1 (2018-11-01)
Fix reindexObjectSecurity patch to account for special handling of system roles in allowedRolesAndUsers indexer. [lgraf]
2.3.0 (2018-10-18)
Patch reindexObjectSecurity to optimize indexing of large trees. [lgraf]
Fix diff for items with a modification date that has no fractional seconds. [buchi]
Provide an error message when raising an AttributeError in SolrDocument. [buchi]
2.2.0 (2018-09-21)
Add option to synchronize Solr with catalog in maintenance view. [buchi]
2.1.2 (2018-05-15)
Set Content-Type ‘application/x-www-form-urlencoded’ for requests to /update/extract endpoint to ensure compatibility with Solr 7.3. [lgraf]
Maintenance view: Allow reindexing without dooming the transaction. [lgraf]
2.1.1 (2018-02-20)
Make sure we do not pass invalid dates to Solr. [buchi]
2.1.0 (2018-02-20)
Declare dependency on plone.namedfile[blobs]. [lgraf]
Add diff function to maintenance view to compare UIDs in portal catalog with UIDs in Solr. [buchi]
Fix security filter: Properly quote roles/users with spaces. [buchi]
Add option to reindex Solr from a catalog query. [buchi]
2.0.0 (2018-02-15)
Complete rewrite. Not compatible with previous versions. [buchi]
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
