No project description provided

Project description

tantivy-py

Python bindings for Tantivy the full-text search engine library written in Rust.

Installation

The bindings can be installed using from pypi using pip:

pip install tantivy

If no binary wheel is present for your operating system the bindings will be build from source, this means that Rust needs to be installed before building can succeed.

Note that the bindings are using PyO3, which only supports python3.

Development

For compiling Python module:

# create virtual env
python -m venv .venv
source .venv/bin/activate

# install maturin, the build tool for PyO3
pip install maturin

# compile and install python module in venv
maturin develop

Setting up a development environment can be done in a virtual environment using nox or using local packages using the provided Makefile.

For the nox setup install the virtual environment and build the bindings using:

python3 -m pip install nox
nox

For the Makefile based setup run:

make

Running the tests is done using:

make test

Usage

The Python bindings have a similar API to Tantivy. To create a index first a schema needs to be built. After that documents can be added to the index and a reader can be created to search the index.

Building an index and populating it

import tantivy

# Declaring our schema.
schema_builder = tantivy.SchemaBuilder()
schema_builder.add_text_field("title", stored=True)
schema_builder.add_text_field("body", stored=True)
schema_builder.add_integer_field("doc_id",stored=True)
schema = schema_builder.build()

# Creating our index (in memory)
index = tantivy.Index(schema)

To have a persistent index, use the path parameter to store the index on the disk, e.g:

index = tantivy.Index(schema, path=os.getcwd() + '/index')

By default, tantivy offers the following tokenizers which can be used in tantivy-py:

default default is the tokenizer that will be used if you do not assign a specific tokenizer to your text field. It will chop your text on punctuation and whitespaces, removes tokens that are longer than 40 chars, and lowercase your text.
raw Does not actual tokenizer your text. It keeps it entirely unprocessed. It can be useful to index uuids, or urls for instance.
en_stem

In addition to what default does, the en_stem tokenizer also apply stemming to your tokens. Stemming consists in trimming words to remove their inflection. This tokenizer is slower than the default one, but is recommended to improve recall.

to use the above tokenizers, simply provide them as a parameter to add_text_field. e.g.

schema_builder.add_text_field("body",  stored=True,  tokenizer_name='en_stem')

Adding one document.

writer = index.writer()
writer.add_document(tantivy.Document(
	doc_id=1,
    title=["The Old Man and the Sea"],
    body=["""He was an old man who fished alone in a skiff in the Gulf Stream and he had gone eighty-four days now without taking a fish."""],
))
# ... and committing
writer.commit()

Building and Executing Queries

First you need to get a searcher for the index

# Reload the index to ensure it points to the last commit.
index.reload()
searcher = index.searcher()

Then you need to get a valid query object by parsing your query on the index.

query = index.parse_query("fish days", ["title", "body"])
(best_score, best_doc_address) = searcher.search(query, 3).hits[0]
best_doc = searcher.doc(best_doc_address)
assert best_doc["title"] == ["The Old Man and the Sea"]
print(best_doc)

Valid Query Formats

tantivy-py supports the query language used in tantivy. Some basic query Formats.

AND and OR conjunctions.

query = index.parse_query('(Old AND Man) OR Stream', ["title", "body"])
(best_score, best_doc_address) = searcher.search(query, 3).hits[0]
best_doc = searcher.doc(best_doc_address)

+(includes) and -(excludes) operators.

query = index.parse_query('+Old +Man chef -fished', ["title", "body"])
(best_score, best_doc_address) = searcher.search(query, 3).hits[0]
best_doc = searcher.doc(best_doc_address)

Note: in a query like above, a word with no +/- acts like an OR.

phrase search.

query = index.parse_query('"eighty-four days"', ["title", "body"])
(best_score, best_doc_address) = searcher.search(query, 3).hits[0]
best_doc = searcher.doc(best_doc_address)

integer search

query = index.parse_query('"eighty-four days"', ["doc_id"])
(best_score, best_doc_address) = searcher.search(query, 3).hits[0]
best_doc = searcher.doc(best_doc_address)

Note: for integer search, the integer field should be indexed.

For more possible query formats and possible query options, see Tantivy Query Parser Docs.

Project details

Release history Release notifications | RSS feed

0.22.0

May 5, 2024

This version

0.21.0

Nov 21, 2023

0.20.1

Sep 11, 2023

0.13.2

Oct 11, 2020

0.13.1-rc.1 pre-release

Sep 20, 2020

0.12.0-rc.2 pre-release

Apr 22, 2020

0.12.0-rc.1 pre-release

Apr 19, 2020

0.11.0-rc.8 pre-release

Jan 25, 2020

0.11.0-rc.7 pre-release

Jan 6, 2020

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tantivy-0.21.0.tar.gz (52.6 kB view details)

Uploaded Nov 21, 2023 Source

Built Distributions

tantivy-0.21.0-cp312-none-win_amd64.whl (2.3 MB view details)

Uploaded Nov 21, 2023 CPython 3.12 Windows x86-64

tantivy-0.21.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.2 MB view details)

Uploaded Nov 21, 2023 CPython 3.12 manylinux: glibc 2.17+ x86-64

tantivy-0.21.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.0 MB view details)

Uploaded Nov 21, 2023 CPython 3.12 manylinux: glibc 2.17+ ARM64

tantivy-0.21.0-cp312-cp312-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl (5.8 MB view details)

Uploaded Nov 21, 2023 CPython 3.12 macOS 10.9+ universal2 (ARM64, x86-64) macOS 10.9+ x86-64 macOS 11.0+ ARM64

tantivy-0.21.0-cp312-cp312-macosx_10_7_x86_64.whl (3.1 MB view details)

Uploaded Nov 21, 2023 CPython 3.12 macOS 10.7+ x86-64

tantivy-0.21.0-cp311-none-win_amd64.whl (2.3 MB view details)

Uploaded Nov 21, 2023 CPython 3.11 Windows x86-64

tantivy-0.21.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.2 MB view details)

Uploaded Nov 21, 2023 CPython 3.11 manylinux: glibc 2.17+ x86-64

tantivy-0.21.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.0 MB view details)

Uploaded Nov 21, 2023 CPython 3.11 manylinux: glibc 2.17+ ARM64

tantivy-0.21.0-cp311-cp311-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl (5.8 MB view details)

Uploaded Nov 21, 2023 CPython 3.11 macOS 10.9+ universal2 (ARM64, x86-64) macOS 10.9+ x86-64 macOS 11.0+ ARM64

tantivy-0.21.0-cp311-cp311-macosx_10_7_x86_64.whl (3.1 MB view details)

Uploaded Nov 21, 2023 CPython 3.11 macOS 10.7+ x86-64

tantivy-0.21.0-cp310-none-win_amd64.whl (2.3 MB view details)

Uploaded Nov 21, 2023 CPython 3.10 Windows x86-64

tantivy-0.21.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.2 MB view details)

Uploaded Nov 21, 2023 CPython 3.10 manylinux: glibc 2.17+ x86-64

tantivy-0.21.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.0 MB view details)

Uploaded Nov 21, 2023 CPython 3.10 manylinux: glibc 2.17+ ARM64

tantivy-0.21.0-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl (5.8 MB view details)

Uploaded Nov 21, 2023 CPython 3.10 macOS 10.9+ universal2 (ARM64, x86-64) macOS 10.9+ x86-64 macOS 11.0+ ARM64

tantivy-0.21.0-cp310-cp310-macosx_10_7_x86_64.whl (3.1 MB view details)

Uploaded Nov 21, 2023 CPython 3.10 macOS 10.7+ x86-64

tantivy-0.21.0-cp39-none-win_amd64.whl (2.3 MB view details)

Uploaded Nov 21, 2023 CPython 3.9 Windows x86-64

tantivy-0.21.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.2 MB view details)

Uploaded Nov 21, 2023 CPython 3.9 manylinux: glibc 2.17+ x86-64

tantivy-0.21.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.0 MB view details)

Uploaded Nov 21, 2023 CPython 3.9 manylinux: glibc 2.17+ ARM64

tantivy-0.21.0-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl (5.8 MB view details)

Uploaded Nov 21, 2023 CPython 3.9 macOS 10.9+ universal2 (ARM64, x86-64) macOS 10.9+ x86-64 macOS 11.0+ ARM64

tantivy-0.21.0-cp39-cp39-macosx_10_7_x86_64.whl (3.1 MB view details)

Uploaded Nov 21, 2023 CPython 3.9 macOS 10.7+ x86-64

tantivy-0.21.0-cp38-none-win_amd64.whl (2.3 MB view details)

Uploaded Nov 21, 2023 CPython 3.8 Windows x86-64

tantivy-0.21.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.2 MB view details)

Uploaded Nov 21, 2023 CPython 3.8 manylinux: glibc 2.17+ x86-64

tantivy-0.21.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.0 MB view details)

Uploaded Nov 21, 2023 CPython 3.8 manylinux: glibc 2.17+ ARM64

tantivy-0.21.0-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl (5.8 MB view details)

Uploaded Nov 21, 2023 CPython 3.8 macOS 10.9+ universal2 (ARM64, x86-64) macOS 10.9+ x86-64 macOS 11.0+ ARM64

tantivy-0.21.0-cp38-cp38-macosx_10_7_x86_64.whl (3.1 MB view details)

Uploaded Nov 21, 2023 CPython 3.8 macOS 10.7+ x86-64

File details

Details for the file tantivy-0.21.0.tar.gz.

File metadata

Download URL: tantivy-0.21.0.tar.gz
Upload date: Nov 21, 2023
Size: 52.6 kB
Tags: Source
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/4.0.2 CPython/3.11.6

File hashes

Hashes for tantivy-0.21.0.tar.gz
Algorithm	Hash digest
SHA256	`fe1ea890b2d924417b93ced6036e5f688e606d7b1af9711a0d8ce826172255c2`
MD5	`ba4e9685787324f8a24ad53f777a30ee`
BLAKE2b-256	`ccbb57cf75af82ee79bd82a0aa0e6fbe7f33b97b8befd6599c023edde3cc37bf`

Algorithm	Hash digest
SHA256	`7aacd0c96b681e6933e276c81cc822b0259c6414088287a4b62c263d529c7109`
MD5	`e6f6c0ac805e288c018f259fa6f05578`
BLAKE2b-256	`2d2b6f42b93f61ef105964f75acb67266782cd7d5ae77c8840bf2c7b8562ddd5`

Algorithm	Hash digest
SHA256	`1050546130ff92618191477e4dea8a8b3a02fb6db83226b21efd2059b153d9fc`
MD5	`ee0d3963105f6df041b7822024df9123`
BLAKE2b-256	`30bf939b9dd9e3107efb54e9458979df09b1a15861e045c722ebdb0ec377851d`

Algorithm	Hash digest
SHA256	`521a61248b97c22adf67c06a02b344bb2b231ecf287d2451a76585fbf52d1225`
MD5	`63958fd4b0c8163cf79f6d010c460b87`
BLAKE2b-256	`f87ffd968b5b59b3cc46621e3d11d2b4099bbcff7f32a6b4b98fe6e828e0c279`

Algorithm	Hash digest
SHA256	`c2342b4819a386ff4424bfefee5965f791a91192f367a62071ca1ab29525c381`
MD5	`0a0f46f32d31aa9c4397d8295c22b146`
BLAKE2b-256	`d8a6b95f8763c075d4e23e0f695aa2ec8c4ea06e0d44ad3ccd772b11c63a6f96`

Algorithm	Hash digest
SHA256	`4639c2b59a818174ad83f4363d4e48092e348908ac1755970fd5352e22917d37`
MD5	`0aab3fd617dae366317eecfe657ba7db`
BLAKE2b-256	`bdcabcafcd70596ae5afef5678dcc3cb0ed10acadc8054821148058f858d4121`

Algorithm	Hash digest
SHA256	`8ae347af20c970262d2207b1c122fb0a702c35ee3e052773e31aae80f32a759a`
MD5	`822c5853d236cf3637ab2633d0009a70`
BLAKE2b-256	`f059a27f108ca4a423e713bbf0f9fad9220c89ddef41d8613e1b90858d6678c1`

Algorithm	Hash digest
SHA256	`c73be00cf41556295c5f088c3b5de3feb7539ee2ba62440d244c13b71c3ff3c6`
MD5	`22a79a8c013e1a38b22bdb270f8b2985`
BLAKE2b-256	`270b42c075a458f7bbde60c82a2072d3aeeebc8cf8d7b6880040a617bc1c482c`

Algorithm	Hash digest
SHA256	`7d3bcf082fc5086b228c5cdebb27460338459b50e64f02e212bb3ca919c74f7f`
MD5	`9720ffd1854f3ea6d5c661a243a4d5ce`
BLAKE2b-256	`28d3c75dfe091b8e41a212f8c0757b68a311b7716ccef7956015d95feb703fc6`

Algorithm	Hash digest
SHA256	`f26bf2b192d09d6c1cea4dbed2455689d6800e0e462041b41c66bb228a46d7ba`
MD5	`66f3b3be92315db5b9bbb2c085376927`
BLAKE2b-256	`11c8b7db6dd8c46ec58533f8ff5872ef23ec7fb85ca1c2c0ce90fa3804bf1e34`

Algorithm	Hash digest
SHA256	`726809ec7bb446d616c9303f9ed5595095f9d738792c98350e3a6ef4128cf189`
MD5	`8b102c57f795479dce408730769a5fe7`
BLAKE2b-256	`06f659461803a4e74d083f77c0439a0b8267c6aa39755d099a9936e311b3cd27`

Algorithm	Hash digest
SHA256	`632f35c083093b964707ef1dac124b006e32a18fce6cd354a28ec5a53275bca8`
MD5	`1647039a41eef072c79f6785e95bcbe2`
BLAKE2b-256	`db147d907ee4aea290a09912e76cc69f538f7069132a7ac5e917189f2ffa12e1`

Algorithm	Hash digest
SHA256	`bd2f9a96ed60aa45ea0b97d553665acc691e994dae6d21a07b0b95b9b0a70be0`
MD5	`2979f1711c67404f9684f2b0cd56a1ab`
BLAKE2b-256	`feef3f2bd6150fdf213c333e941919e8f4f0f515fcdca8197825fea2cd010e0b`

Algorithm	Hash digest
SHA256	`6d6efed2f771eaa3a9ad27540d58236a8357737bc84138269617af317558533a`
MD5	`7079c8806d85f362fbeac59a25c4feb1`
BLAKE2b-256	`351166de01550908eab43977345392f92aee02ff9c3c02a089ed275e8d9abc08`

Algorithm	Hash digest
SHA256	`49a974979cdb0d26c483a18d464fa1e1818aa4e56ec8d2fb14e7e2b7c7df22f6`
MD5	`ecb60b72a4fe7a99c678c0cafa6b936f`
BLAKE2b-256	`95efdd76b6a62a8097452449e013adfd48467b93b89cb440a81dea186f8666a7`

Algorithm	Hash digest
SHA256	`1782689425d76f9be571e080b0d62a08c932d9811205c522a835f1e206005af6`
MD5	`c39a7e464a532c57d78298a8ede627b5`
BLAKE2b-256	`09e0e4bcba98ff157e13e2ca2e46c0276de14981d275fdba2f0c6720922661ef`

Algorithm	Hash digest
SHA256	`72fdb53814ee9f60d0d80d028bc1de753f130712b4c611afcd80a0992a7ac8a7`
MD5	`39e2c2bb0d6fc98777a4817f0b15bafe`
BLAKE2b-256	`5415b49d04d9ee03b316d45876c2be708fe8e1dc68cc5d2d140d100b10f35771`

Algorithm	Hash digest
SHA256	`d274e7828ae7307726ace780019f4ab3c5167e1c31ece1a427983cf2ca1d57bc`
MD5	`005677a2c89013f24dafd33cb50406c9`
BLAKE2b-256	`8661d991df638b46a433655f3cefeea4b267047673bc7d5a5c898c53fd106736`

Algorithm	Hash digest
SHA256	`8e5290d32459e16b496ff0ae95dcb7e41f569b832ce5abf850042a88173c7ca2`
MD5	`d3414199d9cdf8387de23a02e426947c`
BLAKE2b-256	`a7487f12bb8e7446af145a8653418015d5c1adc61abd0c0b18113fc843740c25`

Algorithm	Hash digest
SHA256	`2122f0845c84bf34607edbf283840458c7714c0ab2b6973a56aee8820cae642d`
MD5	`8f3a86c411e8f961ed56fd983ea0a46a`
BLAKE2b-256	`5121cfa591977e92519c138b151898de23f7bbac94dbd1e0d6200d7fcfe1bdd6`

Algorithm	Hash digest
SHA256	`2f904b52bb21cc3cda744e7ec6c06f37b2721f8c00a21b76700e94c0a8a3178f`
MD5	`c899ff6d3db88263f0ca41db9d76ee0b`
BLAKE2b-256	`b7ddbb33934dd4656da831219313fa361521cbb79392fac7009fc7695b1dc54b`

tantivy 0.21.0

Navigation

Verified details

Maintainers

Unverified details

Meta

Project description

tantivy-py

Installation

Development

Usage

Building an index and populating it

Adding one document.

Building and Executing Queries

Valid Query Formats

Project details

Verified details

Maintainers

Unverified details

Meta

Release history Release notifications | RSS feed

Download files

Source Distribution

Built Distributions

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

Algorithm	Hash digest
SHA256	`d15d375c480f1db7791306a7d1b32943c2ce25f92cc96cb74137f45264867928`
MD5	`52e9c2a0735bbcb499d5483121f33e68`
BLAKE2b-256	`ff9ffc4ac775e1f154f930132658e21f84ba5aa07589ac9b425d54f43b147795`

Algorithm	Hash digest
SHA256	`3600c365ddc330d28a129e648187f6623a4cb734f3c4a5f430bdb254e536197b`
MD5	`0fc70c4e4205b9f59241a914877c094b`
BLAKE2b-256	`bfedd3affd477f25f1d38482207bf5a689f424208826d261f6e85b7c0aadfab8`

Algorithm	Hash digest
SHA256	`c08784ce7b668b08bc1eaab747da6dddb4f82acd7c1e4e41fa3a4126c3590dfb`
MD5	`2882f7bd8f8d3b006147c8ba4ff0d912`
BLAKE2b-256	`15a70d3816c89643064503d4b3dbefa548f6b4a7c9bec615a36985c1301b68f2`

Algorithm	Hash digest
SHA256	`e8abe45eb4b2c57254e05ad17386a55b8323e1f23fab4f0f9a9b4618e0251381`
MD5	`1c6b918d5bfe8416ef53788f952188dc`
BLAKE2b-256	`cf08749d8bc4dc5b7912483baea9b359e09be187eb66902ba11fd992f1bad0f4`

Algorithm	Hash digest
SHA256	`ddc58a4e85d0df2893570b4ec52cb7737bf5245a2c20257c88e9bd22600949b9`
MD5	`fd2675b6bad25f6e2def11ce3b6f4ec4`
BLAKE2b-256	`79e2d6e61e5a33b021f96ac55e3bf0d269c7a0965e875e94ad4576c3bc7f7bea`