udata search service
Project description
udata-search-service
A search service for udata. The idea is to have search service separated from the udata MongoDB. The indexation update is made using real-time messages with Kafka.
See the following architecture schema:
Getting started
Start the different services using docker-compose:
docker-compose up
This will start:
- an elasticsearch
- a kafka broker
- a zookeper
- a kafka consumer
- a search app
Initialize the elasticsearch indices
flask init-es
You can feed the elasticsearch by publishing messages to Kafka. Using udata, when you modify objects, indexation messages will be sent and will be consumed by the kafka consumer.
If you want to reindex your local mongo base, you can run:
udata search index
You can query the search service with the search service api, ex: http://localhost:5000/api/1/datasets/?q=toilettes%20à%20rennes
Troubleshooting
- If the elasticsearch service exits with an error 137, it is killed due to out of memory error. You should read the following points.
- If you are short on RAM, you can limit heap memory by setting
ES_JAVA_OPTS=-Xms750m -Xmx750mas environment variable when starting the elasticsearch service. - If you are on MAC and still encounter RAM memory issues, you should increase Docker limit memory to 4GB instead of default 2GB.
- If you are on Linux, you may need to double the vm.max_map_count. You can set it with the following command:
sysctl -w vm.max_map_count=262144. - If you are on Linux, you may encounter permissions issues. You can either create the volume or change the user to the current user using
chown.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file udata-search-service-0.0.0.dev1.tar.gz.
File metadata
- Download URL: udata-search-service-0.0.0.dev1.tar.gz
- Upload date:
- Size: 42.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: python-requests/2.27.1
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 9083f9f30f14af89d0791b39fcdd5dac84b5422c587a383d02096be395788ee9 |
|
| MD5 | 4655d8ab5efba348f40493ac9b7f8020 |
|
| BLAKE2b-256 | 3f86ed0830f5b87fa0ef74d970b681a890bcf3089ba060794963bda246342157 |
File details
Details for the file udata_search_service-0.0.0.dev1-py2.py3-none-any.whl.
File metadata
- Download URL: udata_search_service-0.0.0.dev1-py2.py3-none-any.whl
- Upload date:
- Size: 14.0 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: python-requests/2.27.1
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | d84cac02c1327dcef340392187a1c4af2fe19089f2f8e9e636e561c0ebbecb39 |
|
| MD5 | 916e7576b6c3799217cc9d5c00b87d6b |
|
| BLAKE2b-256 | 58abdfc1df37c5359a187d45111e5a77d04ecd73b48f8d95619e0716b0f15555 |
