Unidecode 0.04.20

Module content

The module exports a function that takes an Unicode object (Python 2.x) or string (Python 3.x) and returns a string (that can be encoded to ASCII bytes in Python 3.x):

>>> from unidecode import unidecode
>>> unidecode(u'ko\u017eu\u0161\u010dek')
'kozuscek'
>>> unidecode(u'30 \U0001d5c4\U0001d5c6/\U0001d5c1')
'30 km/h'
>>> unidecode(u"\u5317\u4EB0")
'Bei Jing '

A utility is also included that allows you to transliterate text from the command line in several ways. Reading from standard input:

$ echo hello | unidecode
hello

from a command line argument:

$ unidecode -c hello
hello

or from a file:

$ unidecode hello.txt
hello

The default encoding used by the utility depends on your system locale. You can specify another encoding with the -e argument. See unidecode –help for a full list of available options.

Requirements

Nothing except Python itself.

You need a Python build with “wide” Unicode characters (also called “UCS-4 build”) in order for unidecode to work correctly with characters outside of Basic Multilingual Plane (BMP). Common characters outside BMP are bold, italic, script, etc. variants of the Latin alphabet intended for mathematical notation. Surrogate pair encoding of “narrow” builds is not supported in unidecode.

If your Python build supports “wide” Unicode the following expression will return True:

>>> import sys
>>> sys.maxunicode > 0xffff
True

See PEP 261 for details regarding support for “wide” Unicode characters in Python.

Installation

To install the latest version of Unidecode from the Python package index, use these commands:

$ pip install unidecode

To install Unidecode from the source distribution and run unit tests, use:

$ python setup.py install
$ python setup.py test

Performance notes

By default, unidecode optimizes for the use case where most of the strings passed to it are already ASCII-only and no transliteration is necessary (this default might change in future versions).

For performance critical applications, two additional functions are exposed:

unidecode_expect_ascii is optimized for ASCII-only inputs (approximately 5 times faster than unidecode_expect_nonascii on 10 character strings, more on longer strings), but slightly slower for non-ASCII inputs.

unidecode_expect_nonascii takes approximately the same amount of time on ASCII and non-ASCII inputs, but is slightly faster for non-ASCII inputs than unidecode_expect_ascii.

Apart from differences in run time, both functions produce identical results. For most users of Unidecode, the difference in performance should be negligible.

Source

You can get the latest development version of Unidecode with:

$ git clone https://www.tablix.org/~avian/git/unidecode.git

There is also an official mirror of this repository on GitHub at https://github.com/avian2/unidecode

Contact

Please send bug reports, patches and suggestions for Unidecode to tomaz.solc@tablix.org.

Alternatively, you can also open a ticket or pull request at https://github.com/avian2/unidecode

Copyright

Original character transliteration tables:

Copyright 2001, Sean M. Burke <sburke@cpan.org>, all rights reserved.

Python code and later additions:

Copyright 2017, Tomaz Solc <tomaz.solc@tablix.org>

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. The programs and documentation in this dist are distributed in the hope that they will be useful, but without any warranty; without even the implied warranty of merchantability or fitness for a particular purpose.