Integrate Elasticsearch DSL with Django REST framework.
Project description
Integrate django-elasticsearch-dsl with Django REST framework in the shortest way possible, with least efforts possible.
Package provides views, filter backends and other handy tools.
You are expected to use django-elasticsearch-dsl for defining your document models.
Prerequisites
Django 1.8, 1.9, 1.10 and 1.11.
Python 2.7, 3.4, 3.5, 3.6
Elasticsearch 2.x, 5.x
Dependencies
django-elasticsearch-dsl
djangorestframework
Installation
Install latest stable version from PyPI:
pip install django-elasticsearch-dsl-drf
or latest stable version from GitHub:
pip install https://github.com/barseghyanartur/django-elasticsearch-dsl-drf/archive/stable.tar.gz
Add rest_framework and django_elasticsearch_dsl to INSTALLED_APPS:
INSTALLED_APPS = ( # ... 'rest_framework', # REST framework 'django_elasticsearch_dsl', # Elasticsearch integration # ... )
Supported lookups
Features/issues marked with plus (+) are implemented/solved.
Features/issues marked with minus (-) are yet to be implemented.
Native
The following native (to Elasticsearch) filters/lookups are implemented:
+ ``term``
+ ``terms``
+ ``range``
+ ``exists``
+ ``prefix``
+ ``wildcard``
- ``regexp``
- ``fuzzy``
- ``type``
- ``ids``
Functional
The following functional (non-native to Elasticsearch, but common in Django) filters/lookups are implemented:
+ ``contains``
+ ``in``
- ``gt``
- ``gte``
- ``lt``
- ``lte``
+ ``contains``
+ ``startswith``
+ ``endswith``
+ ``isnull``
+ ``exclude``
Usage examples
See the example project for sample models/views/serializers.
Basic Django REST framework integration example
Sample basic example models
books/models.py:
class Publisher(models.Model):
"""Publisher."""
name = models.CharField(max_length=30)
address = models.CharField(max_length=50)
city = models.CharField(max_length=60)
state_province = models.CharField(max_length=30)
country = models.CharField(max_length=50)
website = models.URLField()
class Meta(object):
"""Meta options."""
ordering = ["id"]
def __str__(self):
return self.name
Sample basic example document
search_indexes/documents/publisher.py:
from django_elasticsearch_dsl import DocType, Index, fields
from elasticsearch_dsl import analyzer
from books.models import Publisher
# Name of the Elasticsearch index
PUBLISHER_INDEX = Index('publisher')
# See Elasticsearch Indices API reference for available settings
PUBLISHER_INDEX.settings(
number_of_shards=1,
number_of_replicas=1
)
@PUBLISHER_INDEX.doc_type
class PublisherDocument(DocType):
"""Publisher Elasticsearch document."""
id = fields.IntegerField(attr='id')
name = fields.StringField(
fields={
'raw': fields.StringField(
analyzer='keyword'
)
}
)
address = fields.StringField(
fields={
'raw': fields.StringField(
analyzer='keyword'
)
}
)
city = fields.StringField(
fields={
'raw': fields.StringField(
analyzer='keyword'
)
}
)
state_province = fields.StringField(
fields={
'raw': fields.StringField(
analyzer='keyword'
)
}
)
country = fields.StringField(
fields={
'raw': fields.StringField(
analyzer='keyword'
)
}
)
website = fields.StringField(
fields={
'raw': fields.StringField(
analyzer='keyword'
)
}
)
class Meta(object):
"""Meta options."""
model = Publisher # The model associate with this DocType
Sample basic example serializer
search_indexes/serializers.py:
import json
from rest_framework import serializers
class PublisherDocumentSerializer(serializers.Serializer):
"""Serializer for Publisher document."""
id = serializers.IntegerField(read_only=True)
name = serializers.CharField(read_only=True)
address = serializers.CharField(read_only=True)
city = serializers.CharField(read_only=True)
state_province = serializers.CharField(read_only=True)
country = serializers.CharField(read_only=True)
website = serializers.CharField(read_only=True)
class Meta(object):
"""Meta options."""
fields = (
'id',
'name',
'address',
'city',
'state_province',
'country',
'website',
)
read_only_fields = fields
Sample basic example view
search_indexes/views.py:
from django_elasticsearch_dsl_drf.filter_backends import (
FilteringFilterBackend,
OrderingFilterBackend,
SearchFilterBackend,
)
from django_elasticsearch_dsl_drf.views import BaseDocumentViewSet
# Example app models
from search_indexes.documents.publisher import PublisherDocument
from search_indxes.serializers import PublisherDocumentSerializer
class PublisherDocumentView(BaseDocumentViewSet):
"""The PublisherDocument view."""
document = PublisherDocument
serializer_class = PublisherDocumentSerializer
lookup_field = 'id'
filter_backends = [
FilteringFilterBackend,
OrderingFilterBackend,
SearchFilterBackend,
]
# Define search fields
search_fields = (
'name',
'address',
'city',
'state_province',
'country',
)
# Define filtering fields
filter_fields = {
'id': None,
'name': 'name.raw',
'city': 'city.raw',
'state_province': 'state_province.raw',
'country': 'country.raw',
}
# Define ordering fields
ordering_fields = {
'id': None,
'name': None,
'city': None,
'country': None,
}
# Specify default ordering
ordering = ('id', 'name',)
Usage simple example
Considering samples above, you should be able to perform the search, sorting and filtering actions described below.
Sample queries
Search
Query param name reserved for search is search. Make sure your models and documents do not have it as a field or attribute.
Multiple search terms are joined with OR.
Let’s assume we have a number of Book items with fields title, description and summary.
Search in all fields
Search in all fields (name, address, city, state_province and country) for word “reilly”.
http://127.0.0.1:8080/search/publisher/?search=reilly
Search a single term on specific field
In order to search in specific field (name) for term “reilly”, add the field name separated with | to the search term.
http://127.0.0.1:8080/search/publisher/?search=name|reilly
Search for multiple terms
In order to search for multiple terms “reilly”, “bloomsbury” add multiple search query params.
http://127.0.0.1:8080/search/publisher/?search=reilly&search=bloomsbury
Search for multiple terms in specific fields
In order to search for multiple terms “reilly”, “bloomsbury” in specific fields add multiple search query params and field names separated with | to each of the search terms.
http://127.0.0.1:8080/search/publisher/?search=name|reilly&search=city|london
Filtering
Let’s assume we have a number of Publisher documents with in cities (Yerevan, Groningen, Amsterdam, London).
Multiple filter terms are joined with AND.
Filter documents by single field
Filter documents by field (city) “yerevan”.
http://127.0.0.1:8080/search/publisher/?city=yerevan
Filter documents by multiple fields
Filter documents by city “Yerevan” and “Groningen”.
http://127.0.0.1:8080/search/publisher/?city__in=yerevan|groningen
Filter document by a single field
Filter documents by (field country) “Armenia”.
http://127.0.0.1:8080/search/publisher/?country=armenia
Filter documents by multiple fields
Filter documents by multiple fields (field city) “Yerevan” and “Amsterdam” with use of functional in query filter.
http://127.0.0.1:8080/search/publisher/?city__in=yerevan|amsterdam
You can achieve the same effect by specifying multiple filters (city) “Yerevan” and “Amsterdam”. Note, that in this case multiple filter terms are joined with OR.
http://127.0.0.1:8080/search/publisher/?city=yerevan&city=amsterdam
If you want the same as above, but joined with AND, add __term to each lookup.
http://127.0.0.1:8080/search/publisher/?city__term=education&city__term=economy
Filter documents by a word part of a single field
Filter documents by a part word part in single field (city) “ondon”.
http://127.0.0.1:8080/search/publisher/?city__wildcard=*ondon
Ordering
The - prefix means ordering should be descending.
Order documents by field (ascending)
Filter documents by field city (ascending).
http://127.0.0.1:8080/search/publisher/?search=country|armenia&ordering=city
Order documents by field (descending)
Filter documents by field country (descending).
http://127.0.0.1:8080/search/publisher/?ordering=-country
Order documents by multiple fields
If you want to order by multiple fields, use multiple ordering query params. In the example below, documents would be ordered first by field country (descending), then by field city (ascending).
http://127.0.0.1:8080/search/publisher/?ordering=-country&ordering=city
Advanced Django REST framework integration example
Sample advanced example models
books/models.py:
import json
from django.conf import settings
from django.db import models
from django.utils.translation import ugettext, ugettext_lazy as _
from six import python_2_unicode_compatible
BOOK_PUBLISHING_STATUS_PUBLISHED = 'published'
BOOK_PUBLISHING_STATUS_NOT_PUBLISHED = 'not_published'
BOOK_PUBLISHING_STATUS_IN_PROGRESS = 'in_progress'
BOOK_PUBLISHING_STATUS_CANCELLED = 'cancelled'
BOOK_PUBLISHING_STATUS_REJECTED = 'rejected'
BOOK_PUBLISHING_STATUS_CHOICES = (
(BOOK_PUBLISHING_STATUS_PUBLISHED, "Published"),
(BOOK_PUBLISHING_STATUS_NOT_PUBLISHED, "Not published"),
(BOOK_PUBLISHING_STATUS_IN_PROGRESS, "In progress"),
(BOOK_PUBLISHING_STATUS_CANCELLED, "Cancelled"),
(BOOK_PUBLISHING_STATUS_REJECTED, "Rejected"),
)
BOOK_PUBLISHING_STATUS_DEFAULT = BOOK_PUBLISHING_STATUS_PUBLISHED
@python_2_unicode_compatible
class Publisher(models.Model):
"""Publisher."""
name = models.CharField(max_length=30)
address = models.CharField(max_length=50)
city = models.CharField(max_length=60)
state_province = models.CharField(max_length=30)
country = models.CharField(max_length=50)
website = models.URLField()
class Meta(object):
"""Meta options."""
ordering = ["id"]
def __str__(self):
return self.name
@python_2_unicode_compatible
class Author(models.Model):
"""Author."""
salutation = models.CharField(max_length=10)
name = models.CharField(max_length=200)
email = models.EmailField()
headshot = models.ImageField(upload_to='authors', null=True, blank=True)
class Meta(object):
"""Meta options."""
ordering = ["id"]
def __str__(self):
return self.name
class Tag(models.Model):
"""Simple tag model."""
title = models.CharField(max_length=255, unique=True)
class Meta(object):
"""Meta options."""
verbose_name = _("Tag")
verbose_name_plural = _("Tags")
def __str__(self):
return self.title
@python_2_unicode_compatible
class Book(models.Model):
"""Book."""
title = models.CharField(max_length=100)
description = models.TextField(null=True, blank=True)
summary = models.TextField(null=True, blank=True)
authors = models.ManyToManyField('books.Author', related_name='books')
publisher = models.ForeignKey(Publisher, related_name='books')
publication_date = models.DateField()
state = models.CharField(max_length=100,
choices=BOOK_PUBLISHING_STATUS_CHOICES,
default=BOOK_PUBLISHING_STATUS_DEFAULT)
isbn = models.CharField(max_length=100, unique=True)
price = models.DecimalField(max_digits=10, decimal_places=2)
pages = models.PositiveIntegerField(default=200)
stock_count = models.PositiveIntegerField(default=30)
tags = models.ManyToManyField('books.Tag',
related_name='books',
blank=True)
class Meta(object):
"""Meta options."""
ordering = ["isbn"]
def __str__(self):
return self.title
@property
def publisher_indexing(self):
"""Publisher for indexing.
Used in Elasticsearch indexing.
"""
if self.publisher is not None:
return self.publisher.name
@property
def tags_indexing(self):
"""Tags for indexing.
Used in Elasticsearch indexing.
"""
return json.dumps([tag.title for tag in self.tags.all()])
Sample advanced example document
search_indexes/documents/book.py:
from django_elasticsearch_dsl import DocType, Index, fields
from elasticsearch_dsl import analyzer
from books.models import Book
# Name of the Elasticsearch index
BOOK_INDEX = Index('book')
# See Elasticsearch Indices API reference for available settings
BOOK_INDEX.settings(
number_of_shards=1,
number_of_replicas=1
)
html_strip = analyzer(
'html_strip',
tokenizer="standard",
filter=["standard", "lowercase", "stop", "snowball"],
char_filter=["html_strip"]
)
@BOOK_INDEX.doc_type
class BookDocument(DocType):
"""Book Elasticsearch document."""
id = fields.IntegerField(attr='id')
title = fields.StringField(
analyzer=html_strip,
fields={
'raw': fields.StringField(
analyzer='keyword'
)
}
)
description = fields.StringField(
analyzer=html_strip,
fields={
'raw': fields.StringField(
analyzer='keyword'
)
}
)
summary = fields.StringField(
analyzer=html_strip,
fields={
'raw': fields.StringField(
analyzer='keyword'
)
}
)
publisher = fields.StringField(
attr='publisher_indexing',
analyzer=html_strip,
fields={
'raw': fields.StringField(
analyzer='keyword'
)
}
)
publication_date = fields.DateField()
state = fields.StringField(
analyzer=html_strip,
fields={
'raw': fields.StringField(
analyzer='keyword'
)
}
)
isbn = fields.StringField(
analyzer=html_strip,
fields={
'raw': fields.StringField(
analyzer='keyword'
)
}
)
price = fields.FloatField()
pages = fields.IntegerField()
stock_count = fields.IntegerField()
tags = fields.StringField(
attr='tags_indexing',
analyzer=html_strip,
fields={
'raw': fields.StringField(
analyzer='keyword',
multi=True
)
},
multi=True
)
class Meta(object):
"""Meta options."""
model = Book # The model associate with this DocType
Sample advanced example serializer
search_indexes/serializers.py:
import json
from rest_framework import serializers
class TagSerializer(serializers.Serializer):
"""Helper serializer for the Tag field of the Book document."""
title = serializers.CharField()
class Meta(object):
"""Meta options."""
fields = ('title',)
read_only_fields = ('title',)
class BookDocumentSerializer(serializers.Serializer):
"""Serializer for the Book document."""
id = serializers.SerializerMethodField()
title = serializers.CharField(read_only=True)
description = serializers.CharField(read_only=True)
summary = serializers.CharField(read_only=True)
publisher = serializers.CharField(read_only=True)
publication_date = serializers.DateField(read_only=True)
state = serializers.CharField(read_only=True)
isbn = serializers.CharField(read_only=True)
price = serializers.FloatField(read_only=True)
pages = serializers.IntegerField(read_only=True)
stock_count = serializers.IntegerField(read_only=True)
tags = serializers.SerializerMethodField()
class Meta(object):
"""Meta options."""
fields = (
'id',
'title',
'description',
'summary',
'publisher',
'publication_date',
'state',
'isbn',
'price',
'pages',
'stock_count',
'tags',
)
read_only_fields = fields
def get_tags(self, obj):
"""Get tags."""
return json.loads(obj.tags)
Sample advanced example view
search_indexes/views.py:
from django_elasticsearch_dsl_drf.constants import (
LOOKUP_FILTER_TERMS,
LOOKUP_FILTER_RANGE,
LOOKUP_FILTER_PREFIX,
LOOKUP_FILTER_WILDCARD,
LOOKUP_QUERY_IN,
LOOKUP_QUERY_EXCLUDE,
)
from django_elasticsearch_dsl_drf.filter_backends import (
FilteringFilterBackend,
OrderingFilterBackend,
SearchFilterBackend,
)
from django_elasticsearch_dsl_drf.views import BaseDocumentViewSet
# Example app models
from search_indexes.documents.book import BookDocument
from search_indxes.serializers import BookDocumentSerializer
class BookDocumentView(BaseDocumentViewSet):
"""The BookDocument view."""
document = BookDocument
serializer_class = BookDocumentSerializer
lookup_field = 'id'
filter_backends = [
FilteringFilterBackend,
OrderingFilterBackend,
SearchFilterBackend,
]
# Define search fields
search_fields = (
'title',
'description',
'summary',
)
# Define filtering fields
filter_fields = {
'id': {
'field': '_id',
'lookups': [
LOOKUP_FILTER_RANGE,
LOOKUP_QUERY_IN,
],
},
'publisher': {
'field': 'publisher.raw',
},
'publication_date': 'publication_date',
'isbn': {
'field': 'isbn.raw',
},
'tags': {
'field': 'tags',
'lookups': [
LOOKUP_FILTER_TERMS,
LOOKUP_FILTER_PREFIX,
LOOKUP_FILTER_WILDCARD,
LOOKUP_QUERY_IN,
LOOKUP_QUERY_EXCLUDE,
],
},
'tags.raw': {
'field': 'tags.raw',
'lookups': [
LOOKUP_FILTER_TERMS,
LOOKUP_FILTER_PREFIX,
LOOKUP_FILTER_WILDCARD,
LOOKUP_QUERY_IN,
LOOKUP_QUERY_EXCLUDE,
],
},
}
# Define ordering fields
ordering_fields = {
'id': 'id',
'title': 'title.raw',
'price': 'price.raw',
'state': 'state.raw',
'publication_date': 'publication_date',
}
# Specify default ordering
ordering = ('id', 'title',)
Usage advanced example
Considering samples above, you should be able to perform the search, sorting and filtering actions described below.
Sample queries
Search
Query param name reserved for search is search. Make sure your models and documents do not have it as a field or attribute.
Multiple search terms are joined with OR.
Let’s assume we have a number of Book items with fields title, description and summary.
Search in all fields
Search in all fields (title, description and summary) for word “education”.
http://127.0.0.1:8080/search/books/?search=education
Search a single term on specific field
In order to search in specific field (title) for term “education”, add the field name separated with | to the search term.
http://127.0.0.1:8080/search/books/?search=title|education
Search for multiple terms
In order to search for multiple terms “education”, “technology” add multiple search query params.
http://127.0.0.1:8080/search/books/?search=education&search=technology
Search for multiple terms on specific fields
In order to search for multiple terms “education”, “technology” in specific fields add multiple search query params and field names separated with | to each of the search terms.
http://127.0.0.1:8080/search/books/?search=title|education&search=summary|technology
Filtering
Let’s assume we have a number of Book documents with the tags (education, politics, economy, biology, climate, environment, internet, technology).
Multiple filter terms are joined with AND.
Filter documents by field
Filter documents by field (state) “published”.
http://127.0.0.1:8080/search/books/?state=published
Filter documents by multiple fields
Filter documents by field (states) “published” and “in_progress”.
http://127.0.0.1:8080/search/books/?state__in=published|in_progress
Filter document by a single field
Filter documents by (field tag) “education”.
http://127.0.0.1:8080/search/books/?tag=education
Filter documents by multiple fields
Filter documents by multiple fields (field tags) “education” and “economy” with use of functional in query filter.
http://127.0.0.1:8080/search/books/?tags__in=education|economy
You can achieve the same effect by specifying multiple fields (tags) “education” and “economy”. Note, that in this case multiple filter terms are joined with OR.
http://127.0.0.1:8080/search/books/?tags=education&tags=economy
If you want the same as above, but joined with AND, add __term to each lookup.
http://127.0.0.1:8080/search/books/?tags__term=education&tags__term=economy
Filter documents by a word part of a single field
Filter documents by a part word part in single field (tags). Word part should match both “technology” and “biology”.
http://127.0.0.1:8080/search/books/?tags__wildcard=*logy
Ordering
The - prefix means ordering should be descending.
Order documents by field (ascending)
Filter documents by field price (ascending).
http://127.0.0.1:8080/search/books/?search=title|lorem&ordering=price
Order documents by field (descending)
Filter documents by field price (descending).
http://127.0.0.1:8080/search/books/?search=title|lorem&ordering=-price
Order documents by multiple fields
If you want to order by multiple fields, use multiple ordering query params. In the example below, documents would be ordered first by field publication_date (descending), then by field price (ascending).
http://127.0.0.1:8080/search/books/?search=title|lorem&ordering=-publication_date&ordering=price
Various handy helpers
More like this
To get more-like-this results on a random registered model, do as follows:
from django_elasticsearch_dsl_drf.helpers import more_like_this
from books.models import Book
book = Book.objects.first()
similar_books = more_like_this(
book,
['title', 'description', 'summary']
)
Testing
Project is covered with tests.
To test with all supported Python/Django versions type:
tox
To test against specific environment, type:
tox -e py36-django110
To test just your working environment type:
./runtests.py
To run a single test in your working environment type:
./runtests.py src/django_elasticsearch_dsl_drf/tests/test_filtering.py
Or:
./manage.py test django_elasticsearch_dsl_drf.tests.test_ordering
It’s assumed that you have all the requirements installed. If not, first install the test requirements:
pip install -r examples/requirements/test.txt
Writing documentation
Keep the following hierarchy.
=====
title
=====
header
======
sub-header
----------
sub-sub-header
~~~~~~~~~~~~~~
sub-sub-sub-header
^^^^^^^^^^^^^^^^^^
sub-sub-sub-sub-header
++++++++++++++++++++++
sub-sub-sub-sub-sub-header
**************************
License
GPL 2.0/LGPL 2.1
Support
For any issues contact me at the e-mail given in the Author section.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for django-elasticsearch-dsl-drf-0.1.5.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | d36ca5e01983b25c1397fa3a97f98a77bcfa78ec3068b80dfefdb2e2d0f71fba |
|
MD5 | f938c36099e051d1d2292fe0dd51fc5a |
|
BLAKE2b-256 | bbe6da2db81e2ebd11b7a3041341516ca85ce082702990a0482a14ef340ddcd3 |
Hashes for django_elasticsearch_dsl_drf-0.1.5-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 65c92d5b40ace1514395a0015ca5dc902b5f9cdaf2cc1def1753195ebf1048f7 |
|
MD5 | c53f44603b3c9164db4ce0a3b77b8da4 |
|
BLAKE2b-256 | 4f26a6a0167c9696c881f75878b95702c7e0e486be28b56369169ca123ab8cb2 |