Some helpful unicode string processing tools in native Python
Project description
cldr-language-helpers
Basically this projects was designed to provide a map of UTF8 Block Ranges to the locale/language code.
It uses cldr.unicode.org project as its reference.
Build
Clone the latest cldr release:
svn co http://www.unicode.org/repos/cldr/tags/release-34/ ./cldr-release-34
Tests
pip install -r requirements/dev.txt
Then run test:
pytest
Tests will also populate the cldr_language_helpers/data directory. Or you can use just run populate tests:
pytest -sm generate
API
from cldr_language_helpers.annotator import StringAnnotator
assert StringAnnotator('123').char_types_by_index == [{'numbers'}, {'numbers'}, {'numbers'}]
assert 'ru_RU' in StringAnnotator('ф').langs_by_index[0]
assert {'ru_RU', 'en_US', 'en', 'ru'}.issubset(StringAnnotator('йцу 123 qwe LOL').all_langs)
stats = StringAnnotator('qwe йцу').lang_stats
assert stats['ru_RU'] == 3
assert stats['ru'] == 3
assert stats['en'] == 3
assert stats['space'] == 1
assert 'en' in StringAnnotator('somesortof123').langs_intersection
assert StringAnnotator('somesortof123').char_types_intersection == set()
assert StringAnnotator('somesortof').char_types_intersection == {'auxiliary', 'main'}
assert StringAnnotator(' ').char_types_intersection == {'space'}
assert StringAnnotator('что-то everything как-то lol !').split_by_lang_intersection() == \
['что-то', ' ', 'everything', ' ', 'как-то', ' ', 'lol', ' ', '!']
assert StringAnnotator('somesortof123').split_by_lang_intersection() == ['somesortof123']
assert StringAnnotator('qwe, 123!!!').split_by_char_type() == \
['qwe', ',', ' ', '123', '!!!']
assert StringAnnotator('йцу 123 qwe LOL').has_langs('ru', 'en')
assert StringAnnotator().has_langs_throughout('ru') is False
assert StringAnnotator('йцу 123').has_langs_throughout('ru') is True
assert StringAnnotator('йцу 123').has_langs_throughout('ru_RU') is True
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
cldr-language-helpers-1.1.1.tar.gz
(313.6 kB
view hashes)
Built Distribution
Close
Hashes for cldr-language-helpers-1.1.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2e49777927efefa7e1f1aa445e2b5532484dee7b92d0bbcb77d960341e138c75 |
|
MD5 | 48074f0c761ae118b0a6ac43c6d0cd68 |
|
BLAKE2b-256 | 780ecb70650872945ba417db6a2ab289f9900d6b999fcaa17fcdae8f9f66c379 |
Close
Hashes for cldr_language_helpers-1.1.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d84f5f6ea5c1aaf6bdfd9725d050c6282922d828a070f5f855fa1c4102a7c20f |
|
MD5 | 8ab1f33c0251dbe1e2f7cbd0bff89e43 |
|
BLAKE2b-256 | 0ab04e1f8027937e9768309c1746dccbfbcdfc89757af6a8aa1f5e12d1ab50fa |