DAWG 0.3.2

DAWG and CompletionDAWG

DAWG and CompletionDAWG are useful when you need fast & memory efficient simple string storage. These classes does not support assigning values to keys.

DAWG and CompletionDAWG constructors accept an iterable with keys:

>>> import dawg
>>> words = [u'foo', u'bar', u'foobar', u'foö', u'bör']
>>> base_dawg = dawg.DAWG(words)
>>> completion_dawg = dawg.CompletionDAWG(words)

It is then possible to check if the key is in a DAWG:

>>> u'foo' in base_dawg
True
>>> u'baz' in completion_dawg
False

It is possible to find all keys that starts with a given prefix in a CompletionDAWG:

>>> completion_dawg.keys(u'foo')
>>> [u'foo', u'foobar']

and to find all prefixes of a given key:

>>> base_dawg.prefixes(u'foobarz')
[u'foo', u'foobar']

It is possible to find all keys similar to a given key (using a one-way char translation table):

>>> replaces = dawg.DAWG.compile_replaces({u'o': u'ö'})
>>> base_dawg.similar_keys(u'foo', replaces)
[u'foo', u'foö']
>>> base_dawg.similar_keys(u'foö', replaces)
[u'foö']
>>> base_dawg.similar_keys(u'bor', replaces)
[u'bör']

BytesDAWG

BytesDAWG is a CompletionDAWG subclass that can store binary data for each key.

BytesDAWG constructor accepts an iterable with (unicode_key, bytes_value) tuples:

>>> data = [(u'key1', b'value1'), (u'key2', b'value2'), (u'key1', b'value3')]
>>> bytes_dawg = dawg.BytesDAWG(data)

There can be duplicate keys; all unique values are stored in this case:

>>> bytes_dawg[u'key1']
[b'value1, b'value3']

For unique keys a list with a single value is returned for consistency:

>>> bytes_dawg[u'key2']
[b'value2']

KeyError is raised for missing keys; use get method if you need a default value instead:

>>> bytes_dawg.get(u'foo', None)
None

BytesDAWG support keys and items methods (they both accept optional key prefix). There is also support for similar_keys, similar_items and similar_item_values methods.

RecordDAWG

RecordDAWG is a BytesDAWG subclass that automatically packs & unpacks the binary data from/to Python objects using struct module from the standard library.

First, you have to define a format of the data. Consult Python docs (http://docs.python.org/library/struct.html#format-strings) for the format string specification.

For example, let’s store 3 short unsigned numbers (in a Big-Endian byte order) as values:

>>> format = ">HHH"

RecordDAWG constructor accepts an iterable with (unicode_key, value_tuple). Let’s create such iterable using zip function:

>>> keys = [u'foo', u'bar', u'foobar', u'foo']
>>> values = [(1, 2, 3), (2, 1, 0), (3, 3, 3), (2, 1, 5)]
>>> data = zip(keys, values)
>>> record_dawg = RecordDAWG(format, data)

As with BytesDAWG, there can be several values for the same key:

>>> record_dawg['foo']
[(1, 2, 3), (2, 1, 5)]
>>> record_dawg['foobar']
[(3, 3, 3)]

IntDAWG

IntDAWG is a {unicode -> int} mapping. It is possible to use RecordDAWG for this, but IntDAWG is natively supported by dawgdic C++ library and so __getitem__ is much faster.

Unlike BytesDAWG and RecordDAWG, IntDAWG doesn’t support having several values for the same key.

IntDAWG constructor accepts an iterable with (unicode_key, integer_value) tuples:

>>> data = [ (u'foo', 1), (u'bar', 2) ]
>>> int_dawg = dawg.IntDAWG(data)

It is then possible to get a value from the IntDAWG:

>>> int_dawg[u'foo']
1

Persistence

All DAWGs support saving/loading and pickling/unpickling.

Write DAWG to a stream:

>>> with open('words.dawg', 'wb') as f:
...     d.write(f)

Save DAWG to a file:

>>> d.save('words.dawg')

Load DAWG from a file:

>>> d = dawg.DAWG()
>>> d.load('words.dawg')

Read DAWG from a stream:

>>> d = dawg.RecordDAWG(format_string)
>>> with open('words.record-dawg', 'rb') as f:
...     d.read(f)

DAWG objects are picklable:

>>> import pickle
>>> data = pickle.dumps(d)
>>> d2 = pickle.loads(data)

How is source code organized

There are 4 folders in repository:

• bench - benchmarks & benchmark data;

• lib - original unmodified dawgdic C++ library and a customized version of libb64 library. They are bundled for easier distribution; if something is have to be fixed in these libraries consider fixing it in the original repositories;

• src - wrapper code; src/dawg.pyx is a wrapper implementation; src/*.pxd files are Cython headers for corresponding C++ headers; src/*.cpp files are the pre-built extension code and shouldn’t be modified directly (they should be updated via update_cpp.sh script).

• tests - the test suite.

Running tests and benchmarks

Make sure tox is installed and run

$ tox

from the source checkout. Tests should pass under python 2.6, 2.7, 3.2 and 3.3.

In order to run benchmarks, type

$ tox -c bench.ini

Authors & Contributors

• Mikhail Korobov <kmike84@gmail.com>

This module is based on dawgdic C++ library by Susumu Yata & contributors.

base64 decoder is based on libb64 by Chris Venter.

0.3.2 (2012-09-24)

• prefixes method for finding all prefixes of a given key.

0.3.1 (2012-09-20)

• bundled dawgdic C++ library is updated to the latest version.

0.3 (2012-09-13)

• similar_keys, similar_items and similar_item_values methods for more permissive lookups (they may be useful e.g. for umlaut handling);

• load method returns self;

• Python 3.3 support.

0.2 (2012-09-08)

Greatly improved memory usage for DAWGs loaded with load method.

There is currently a bug somewhere in a wrapper so DAWGs loaded with read() method or unpickled DAWGs uses 3x-4x memory compared to DAWGs loaded with load() method. load() is fixed in this release but other methods are not.

0.1 (2012-09-08)

Initial release.