Skip to main content

udata search service

Project description

udata-search-service

A search service for udata. The idea is to have search service separated from the udata MongoDB. The indexation update is made using real-time HTTP messages.

See the following architecture schema: Udata Search Service architecture schema

Getting started

You can follow this recommended architecture for your code:

$WORKSPACE
├── fs
├── udata
│   ├── ...
│   └── setup.py
│		└── udata.cfg
├── udata-front
│   ├── ...
│   └── setup.py
└── udata-search-service
    ├── ...
    └── pyproject.toml

Clone the repository:

cd $WORKSPACE
git clone git@github.com:opendatateam/udata-search-service.git

Start the different services using docker-compose:

cd udata-search-service
docker-compose up

This will start:

  • an elasticsearch
  • a search app

Initialize the elasticsearch indices on setup.

# Locally
udata-search-service init-es

# In the docker context
docker-compose run --entrypoint /bin/bash web -c 'udata-search-service init-es'

This will create the following indices:

  • {UDATA_INSTANCE_NAME}-dataset-{yyyy}-{mm}-{dd}-{HH}-{MM}
  • {UDATA_INSTANCE_NAME}-reuse-{yyyy}-{mm}-{dd}-{HH}-{MM}
  • {UDATA_INSTANCE_NAME}-organization-{yyyy}-{mm}-{dd}-{HH}-{MM}

Configure your udata to use the search service, by updating the following variables in your udata.cfg. Ex in local:

    SEARCH_SERVICE_API_URL = 'http://127.0.0.1:5000/api/1/'

Using udata, when you modify objects, indexation messages will be sent to the search app and will be consumed by the API. If you want to reindex your local mongo base in udata, you can run:

cd $WORKSPACE/udata/
source ./venv/bin/activate
udata search index

Make sure to have the corresponding UDATA_INSTANCE_NAME specified in your udata settings.

After a reindexation, you'll need to change the alias by using the following command:

# Locally
udata-search-service set-alias <index-suffix>

# In the docker context
docker-compose run --entrypoint /bin/bash web -c 'udata-search-service set-alias <index-suffix>'

You can query the search service with the search service api, ex: http://localhost:5000/api/1/datasets/?q=toilettes%20à%20rennes

Development

You can create a virtualenv, activate it and install the requirements with the following commands.

python3 -m venv venv
source venv/bin/activate
make deps
make install

You can start the web search service with the following command:

udata-search-service run

Deployment

The project depends on ElasticSearch 7.16.

Elasticsearch requires the Analysis ICU plugin for your specific version. On Debian, you can take a look at these instructions for installation.

Troubleshooting

  • If the elasticsearch service exits with an error 137, it is killed due to out of memory error. You should read the following points.
  • If you are short on RAM, you can limit heap memory by setting ES_JAVA_OPTS=-Xms750m -Xmx750m as environment variable when starting the elasticsearch service.
  • If you are on MAC and still encounter RAM memory issues, you should increase Docker limit memory to 4GB instead of default 2GB.
  • If you are on Linux, you may need to double the vm.max_map_count. You can set it with the following command: sysctl -w vm.max_map_count=262144.
  • If you are on Linux, you may encounter permissions issues. You can either create the volume or change the user to the current user using chown.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

udata-search-service-2.0.1.dev241.tar.gz (41.9 kB view details)

Uploaded Source

Built Distribution

udata_search_service-2.0.1.dev241-py2.py3-none-any.whl (18.4 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file udata-search-service-2.0.1.dev241.tar.gz.

File metadata

File hashes

Hashes for udata-search-service-2.0.1.dev241.tar.gz
Algorithm Hash digest
SHA256 acb795ea5b0f14ae25e032403167a461b83971fb929e9575bde5f2848b292da6
MD5 f4eb69e037152ade9c69e1529f076744
BLAKE2b-256 e29977e489723448f5db0d59cc1fe8f424c4bfc67352f91e61707a0d75ca14c2

See more details on using hashes here.

File details

Details for the file udata_search_service-2.0.1.dev241-py2.py3-none-any.whl.

File metadata

File hashes

Hashes for udata_search_service-2.0.1.dev241-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 aa3d186f2f36c80b6fe6a62ca05f91c273e82888b5ab7b8f770cab05e98c6462
MD5 468214a56c10f0522ed5ee074a1db7e1
BLAKE2b-256 d9f866bf7c6cb43b13f07c7b93771a9dfd5f77ecf5998e6a503af581773792da

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page