Fast multi-keyword search engine for text strings

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scoder from gravatar.com scoder

Unverified details

These details have not been verified by PyPI
Project links
Meta
Classifiers

Project description

What is Acora?

Acora is ‘fgrep’ for Python, a fast multi-keyword text search engine.

Based on a set of keywords and the Aho-Corasick algorithm, it generates a search automaton and runs it over string input, either unicode or bytes.

Acora comes with both a pure Python implementation and a fast binary module written in Cython. However, note that the current construction algorithm is not suitable for really large sets of keywords (i.e. more than a couple of thousand).

You can find the latest source code on github.

To report a bug or request new features, use the github bug tracker. Please try to provide a short test case that reproduces the problem without requiring too much experimentation or large amounts of data. The easier it is to reproduce the problem, the easier it is to solve it.

Features

  • works with unicode strings and byte strings

  • about 2-3x as fast as Python’s regular expression engine for most input

  • finds overlapping matches, i.e. all matches of all keywords

  • support for case insensitive search (~10x as fast as ‘re’)

  • frees the GIL while searching

  • additional (slow but short) pure Python implementation

  • support for Python 2.5+ and 3.x

  • support for searching in files

  • permissive BSD license

How do I use it?

Import the package:

>>> from acora import AcoraBuilder

Collect some keywords:

>>> builder = AcoraBuilder('ab', 'bc', 'de')
>>> builder.add('a', 'b')

Or:

>>> builder.update(['a', 'b'])  # new in version 2.0

Generate the Acora search engine for the current keyword set:

>>> ac = builder.build()

Search a string for all occurrences:

>>> ac.findall('abc')
[('a', 0), ('ab', 0), ('b', 1), ('bc', 1)]
>>> ac.findall('abde')
[('a', 0), ('ab', 0), ('b', 1), ('de', 2)]

Iterate over the search results as they come in:

>>> for kw, pos in ac.finditer('abde'):
...     print("%2s[%d]" % (kw, pos))
 a[0]
ab[0]
 b[1]
de[2]

Acora also has direct support for parsing files (in binary mode):

>>> keywords = ['Import', 'FAQ', 'Acora', 'NotHere'.upper()]

>>> builder = AcoraBuilder([s.encode('ascii') for s in keywords])
>>> ac = builder.build()

>>> found = set(kw for kw, pos in ac.filefind('README.rst'))
>>> len(found)
3

>>> sorted(str(s.decode('ascii')) for s in found)
['Acora', 'FAQ', 'Import']

FAQs and recipes

  1. How do I run a greedy search for the longest matching keywords?

    >>> builder = AcoraBuilder('a', 'ab', 'abc')
>>> ac = builder.build()

>>> for kw, pos in ac.finditer('abbabc'):
...     print(kw)
a
ab
a
ab
abc

>>> from itertools import groupby
>>> from operator import itemgetter

>>> def longest_match(matches):
...     for pos, match_set in groupby(matches, itemgetter(1)):
...         yield max(match_set)

>>> for kw, pos in longest_match(ac.finditer('abbabc')):
...     print(kw)
ab
abc

    Note that this recipe assumes search terms that do not have inner overlaps apart from their prefix.

  2. How do I parse line-by-line with arbitrary line endings?

    >>> def group_by_lines(s, *keywords):
...     builder = AcoraBuilder('\r', '\n', *keywords)
...     ac = builder.build()
...
...     current_line_matches = []
...     last_ending = None
...
...     for kw, pos in ac.finditer(s):
...         if kw in '\r\n':
...             if last_ending == '\r' and kw == '\n':
...                 continue # combined CRLF
...             yield tuple(current_line_matches)
...             del current_line_matches[:]
...             last_ending = kw
...         else:
...             last_ending = None
...             current_line_matches.append(kw)
...     yield tuple(current_line_matches)

>>> kwds = ['ab', 'bc', 'de']
>>> for matches in group_by_lines('a\r\r\nbc\r\ndede\n\nab', *kwds):
...     print(matches)
()
()
('bc',)
('de', 'de')
()
('ab',)

  3. How do I find whole lines that contain keywords, as fgrep does?

    >>> def match_lines(s, *keywords):
...     builder = AcoraBuilder('\r', '\n', *keywords)
...     ac = builder.build()
...
...     line_start = 0
...     matches = False
...     for kw, pos in ac.finditer(s):
...         if kw in '\r\n':
...             if matches:
...                  yield s[line_start:pos]
...                  matches = False
...             line_start = pos + 1
...         else:
...             matches = True
...     if matches:
...         yield s[line_start:]

>>> kwds = ['x', 'de', '\nstart']
>>> text = 'a line with\r\r\nsome text\r\ndede\n\nab\n start 1\nstart\n'
>>> for line in match_lines(text, *kwds):
...     print(line)
some text
dede
start

Changelog

  • 2.5 [2024-09-14]

    • Update to work with CPython 3.13 by building with Cython 3.0.11.

  • 2.4 [2023-09-17]

    • Update to work with CPython 3.12 by building with Cython 3.0.2.

  • 2.3 [2021-03-27]

    • Update to work with CPython 3.9 by building with Cython 0.29.22.

  • 2.2 [2018-08-16]

    • Update to work with CPython 3.7 by building with Cython 0.29.

  • 2.1 [2017-12-15]

    • fix handling of empty engines (Github issue #18)

  • 2.0 [2016-03-17]

    • rewrite of the construction algorithm to speed it up and save memory

  • 1.9 [2015-10-10]

    • recompiled with Cython 0.23.4 for better compatibility with recent Python versions.

  • 1.8 [2014-02-12]

    • pickle support for the pre-built search engines

    • performance optimisations in builder

    • Unicode parsing is optimised for Python 3.3 and later

    • no longer recompiles sources when Cython is installed, unless --with-cython option is passed to setup.py (requires Cython 0.20+)

    • build failed with recent Cython versions

    • built using Cython 0.20.1

  • 1.7 [2011-08-24]

    • searching binary strings for byte values > 127 was broken

    • built using Cython 0.15+

  • 1.6 [2011-07-24]

    • substantially faster automaton building

    • no longer includes .hg repo in source distribution

    • built using Cython 0.15 (rc0)

  • 1.5 [2011-01-24]

    • Cython compiled NFA-2-DFA construction runs substantially faster

    • always build extension modules even if Cython is not installed

    • --no-compile switch in setup.py to prevent extension module building

    • built using Cython 0.14.1 (rc2)

  • 1.4 [2009-02-10]

    • minor speed-up in inner search engine loop

    • some code cleanup

    • built using Cython 0.12.1 (final)

  • 1.3 [2009-01-30]

    • major fix for file search

    • built using Cython 0.12.1 (beta0)

  • 1.2 [2009-01-30]

    • deep-copy support for AcoraBuilder class

    • doc/test fixes

    • include .hg repo in source distribution

    • built using Cython 0.12.1 (beta0)

  • 1.1 [2009-01-29]

    • doc updates

    • some cleanup

    • built using Cython 0.12.1 (beta0)

  • 1.0 [2009-01-29]

    • initial release

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scoder from gravatar.com scoder

Unverified details

These details have not been verified by PyPI
Project links
Meta
Classifiers

Release history Release notifications | RSS feed

This version

2.5

2.4

2.3

2.2

2.1

2.0

1.9

1.8

1.7

1.6

1.5

1.4

1.3

1.2

1.1

1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

acora-2.5.tar.gz (268.9 kB view hashes)

Uploaded Source

Built Distributions

acora-2.5-cp313-cp313-win_amd64.whl (134.4 kB view hashes)

Uploaded CPython 3.13 Windows x86-64

acora-2.5-cp313-cp313-musllinux_1_1_x86_64.whl (184.2 kB view hashes)

Uploaded CPython 3.13 musllinux: musl 1.1+ x86-64

acora-2.5-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_28_x86_64.whl (181.5 kB view hashes)

Uploaded CPython 3.13 manylinux: glibc 2.17+ x86-64 manylinux: glibc 2.28+ x86-64

acora-2.5-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_28_aarch64.whl (168.7 kB view hashes)

Uploaded CPython 3.13 manylinux: glibc 2.17+ ARM64 manylinux: glibc 2.28+ ARM64

acora-2.5-cp313-cp313-macosx_12_0_universal2.whl (299.2 kB view hashes)

Uploaded CPython 3.13 macOS 12.0+ universal2 (ARM64, x86-64)

acora-2.5-cp312-cp312-win_amd64.whl (135.5 kB view hashes)

Uploaded CPython 3.12 Windows x86-64

acora-2.5-cp312-cp312-musllinux_1_1_x86_64.whl (186.8 kB view hashes)

Uploaded CPython 3.12 musllinux: musl 1.1+ x86-64

acora-2.5-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_28_x86_64.whl (182.7 kB view hashes)

Uploaded CPython 3.12 manylinux: glibc 2.17+ x86-64 manylinux: glibc 2.28+ x86-64

acora-2.5-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_28_aarch64.whl (169.7 kB view hashes)

Uploaded CPython 3.12 manylinux: glibc 2.17+ ARM64 manylinux: glibc 2.28+ ARM64

acora-2.5-cp312-cp312-macosx_12_0_universal2.whl (300.3 kB view hashes)

Uploaded CPython 3.12 macOS 12.0+ universal2 (ARM64, x86-64)

acora-2.5-cp311-cp311-win_amd64.whl (135.1 kB view hashes)

Uploaded CPython 3.11 Windows x86-64

acora-2.5-cp311-cp311-musllinux_1_1_x86_64.whl (193.5 kB view hashes)

Uploaded CPython 3.11 musllinux: musl 1.1+ x86-64

acora-2.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_28_x86_64.whl (190.3 kB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64 manylinux: glibc 2.28+ x86-64

acora-2.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (178.2 kB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64 manylinux: glibc 2.24+ x86-64

acora-2.5-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_24_aarch64.whl (156.8 kB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.17+ ARM64 manylinux: glibc 2.24+ ARM64

acora-2.5-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.manylinux_2_24_i686.whl (179.2 kB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.24+ i686 manylinux: glibc 2.5+ i686

acora-2.5-cp311-cp311-macosx_12_0_universal2.whl (298.1 kB view hashes)

Uploaded CPython 3.11 macOS 12.0+ universal2 (ARM64, x86-64)

acora-2.5-cp310-cp310-win_amd64.whl (134.6 kB view hashes)

Uploaded CPython 3.10 Windows x86-64

acora-2.5-cp310-cp310-musllinux_1_1_x86_64.whl (193.3 kB view hashes)

Uploaded CPython 3.10 musllinux: musl 1.1+ x86-64

acora-2.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_28_x86_64.whl (190.8 kB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64 manylinux: glibc 2.28+ x86-64

acora-2.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (178.0 kB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64 manylinux: glibc 2.24+ x86-64

acora-2.5-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_24_aarch64.whl (157.8 kB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARM64 manylinux: glibc 2.24+ ARM64

acora-2.5-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_24_i686.whl (180.7 kB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.24+ i686 manylinux: glibc 2.5+ i686

acora-2.5-cp310-cp310-macosx_12_0_x86_64.whl (157.7 kB view hashes)

Uploaded CPython 3.10 macOS 12.0+ x86-64

acora-2.5-cp39-cp39-win_amd64.whl (134.9 kB view hashes)

Uploaded CPython 3.9 Windows x86-64

acora-2.5-cp39-cp39-musllinux_1_1_x86_64.whl (194.0 kB view hashes)

Uploaded CPython 3.9 musllinux: musl 1.1+ x86-64

acora-2.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_28_x86_64.whl (190.9 kB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64 manylinux: glibc 2.28+ x86-64

acora-2.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (178.6 kB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64 manylinux: glibc 2.24+ x86-64

acora-2.5-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_24_aarch64.whl (158.1 kB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64 manylinux: glibc 2.24+ ARM64

acora-2.5-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl (182.9 kB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.5+ x86-64

acora-2.5-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.whl (172.1 kB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.5+ i686

acora-2.5-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.manylinux_2_24_i686.whl (181.6 kB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.24+ i686 manylinux: glibc 2.5+ i686

acora-2.5-cp39-cp39-macosx_12_0_x86_64.whl (158.2 kB view hashes)

Uploaded CPython 3.9 macOS 12.0+ x86-64

acora-2.5-cp38-cp38-win_amd64.whl (135.2 kB view hashes)

Uploaded CPython 3.8 Windows x86-64

acora-2.5-cp38-cp38-musllinux_1_1_x86_64.whl (195.6 kB view hashes)

Uploaded CPython 3.8 musllinux: musl 1.1+ x86-64

acora-2.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_28_x86_64.whl (193.1 kB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64 manylinux: glibc 2.28+ x86-64

acora-2.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (179.5 kB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64 manylinux: glibc 2.24+ x86-64

acora-2.5-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_24_aarch64.whl (159.2 kB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARM64 manylinux: glibc 2.24+ ARM64

acora-2.5-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (182.6 kB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.5+ x86-64

acora-2.5-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.whl (171.7 kB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.5+ i686

acora-2.5-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.manylinux_2_24_i686.whl (181.7 kB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.24+ i686 manylinux: glibc 2.5+ i686

acora-2.5-cp38-cp38-macosx_12_0_x86_64.whl (157.6 kB view hashes)

Uploaded CPython 3.8 macOS 12.0+ x86-64

acora-2.5-cp37-cp37m-win_amd64.whl (133.7 kB view hashes)

Uploaded CPython 3.7m Windows x86-64

acora-2.5-cp37-cp37m-musllinux_1_1_x86_64.whl (191.6 kB view hashes)

Uploaded CPython 3.7m musllinux: musl 1.1+ x86-64

acora-2.5-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_28_x86_64.whl (192.6 kB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64 manylinux: glibc 2.28+ x86-64

acora-2.5-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (180.0 kB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64 manylinux: glibc 2.24+ x86-64

acora-2.5-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_24_aarch64.whl (160.0 kB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.17+ ARM64 manylinux: glibc 2.24+ ARM64

acora-2.5-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (190.8 kB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.5+ x86-64

acora-2.5-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.whl (177.4 kB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.5+ i686

acora-2.5-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.manylinux_2_24_i686.whl (184.3 kB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.24+ i686 manylinux: glibc 2.5+ i686

acora-2.5-cp37-cp37m-macosx_12_0_x86_64.whl (156.0 kB view hashes)

Uploaded CPython 3.7m macOS 12.0+ x86-64

acora-2.5-cp36-cp36m-win_amd64.whl (150.1 kB view hashes)

Uploaded CPython 3.6m Windows x86-64

acora-2.5-cp36-cp36m-musllinux_1_1_x86_64.whl (185.6 kB view hashes)

Uploaded CPython 3.6m musllinux: musl 1.1+ x86-64

acora-2.5-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_28_x86_64.whl (183.8 kB view hashes)

Uploaded CPython 3.6m manylinux: glibc 2.17+ x86-64 manylinux: glibc 2.28+ x86-64

acora-2.5-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (174.4 kB view hashes)

Uploaded CPython 3.6m manylinux: glibc 2.17+ x86-64 manylinux: glibc 2.24+ x86-64

acora-2.5-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (183.0 kB view hashes)

Uploaded CPython 3.6m manylinux: glibc 2.5+ x86-64

acora-2.5-cp36-cp36m-manylinux_2_5_i686.manylinux1_i686.whl (171.5 kB view hashes)

Uploaded CPython 3.6m manylinux: glibc 2.5+ i686

acora-2.5-cp36-cp36m-manylinux_2_5_i686.manylinux1_i686.manylinux_2_24_i686.whl (178.2 kB view hashes)

Uploaded CPython 3.6m manylinux: glibc 2.24+ i686 manylinux: glibc 2.5+ i686

acora-2.5-cp36-cp36m-macosx_12_0_x86_64.whl (152.7 kB view hashes)

Uploaded CPython 3.6m macOS 12.0+ x86-64

acora-2.5-cp35-cp35m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (179.9 kB view hashes)

Uploaded CPython 3.5m manylinux: glibc 2.5+ x86-64

acora-2.5-cp35-cp35m-manylinux_2_5_i686.manylinux1_i686.whl (169.2 kB view hashes)

Uploaded CPython 3.5m manylinux: glibc 2.5+ i686

acora-2.5-cp27-cp27mu-manylinux_2_5_x86_64.manylinux1_x86_64.whl (176.4 kB view hashes)

Uploaded CPython 2.7mu manylinux: glibc 2.5+ x86-64

acora-2.5-cp27-cp27mu-manylinux_2_5_i686.manylinux1_i686.whl (160.5 kB view hashes)

Uploaded CPython 2.7mu manylinux: glibc 2.5+ i686

acora-2.5-cp27-cp27m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (178.9 kB view hashes)

Uploaded CPython 2.7m manylinux: glibc 2.5+ x86-64

acora-2.5-cp27-cp27m-manylinux_2_5_i686.manylinux1_i686.whl (162.7 kB view hashes)

Uploaded CPython 2.7m manylinux: glibc 2.5+ i686

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page