Archiving, indexing and search for AMQP messages.
Project description
This package is an add-on application for gocept.amqprun that provides three features:
Archive queue messages by writing them to the filesystesm
Index those messages using ElasticSearch
Search for messages with a HTML/JavaScript GUI
Installation
gocept.amqparchive requires an ElasticSearch server. To set up the archive and index portion, add something like this to your gocept.amqprun configuration:
<eventlog>...</eventlog> <amqp-server>...</amqp-server> <worker> amount 2 component-configuration site.zcml </worker> <settings> gocept.amqparchive.elastic_hostname localhost:9200 </settings>
and in site.zcml:
<configure xmlns="http://namespaces.zope.org/zope" xmlns:amqp="http://namespaces.gocept.com/amqp"> <include package="gocept.amqprun" /> <include package="gocept.amqparchive" /> <amqp:writefiles routing_key="key.one key.two key.three" queue_name="archive" directory="/path/to/archive" pattern="{routing_key}/{date}/{msgid}-{unique}.xml" /> </configure>
The HTML/JavaScript GUI expects /elasticsearch to proxy to the ElasticSearch server, and /messages to point to the archive directory (/path/to/archive in our example). Here’s an nginx config snippet:
http { upstream elasticsearch { server localhost:9200; } server { listen localhost:8080; location /search/ { alias /path/to/gocept.amqparchive.egg/gocept/amqparchive/browser/; index index.html; } location /elasticsearch/ { proxy_pass http://elasticsearch/; } location /messages/ { alias /path/to/archive/; autoindex on; } }
Development
The source code is available in the mercurial repository at https://code.gocept.com/hg/public/gocept.amqparchive
Please report any bugs you find at https://projects.gocept.com/projects/projects/gocept-amqparchive/issues
Changelog
1.2.3 (2012-04-18)
Catch connection errors to ElasticSearch so they don’t break the normal message handling. Messages that have not been index due to this can still be indexed later on via reindex_directory (#9363).
1.2.2 (2012-03-29)
Make amqp server configurable for tests.
1.2.1 (2012-02-22)
Switch to plone.testing.
1.2.0 (2011-08-23)
Transform the XML body into a nested dict so ElasticSearch can index the fields individually.
1.1.0 (2011-08-23)
Add command-line script reindex_directory.
1.0.0 (2011-08-22)
first release.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.