Unidecode 1.3.0

Module content

This library contains a function that takes a string object, possibly containing non-ASCII characters, and returns a string that can be safely encoded to ASCII:

>>> from unidecode import unidecode
>>> unidecode('kožušček')
'kozuscek'
>>> unidecode('30 \U0001d5c4\U0001d5c6/\U0001d5c1')
'30 km/h'
>>> unidecode('\u5317\u4EB0')
'Bei Jing '

You can also specify an errors argument to unidecode() that determines what Unidecode does with characters that are not present in its transliteration tables. The default is 'ignore' meaning that Unidecode will ignore those characters (replace them with an empty string). 'strict' will raise a UnidecodeError. The exception object will contain an index attribute that can be used to find the offending character. 'replace' will replace them with '?' (or another string, specified in the replace_str argument). 'preserve' will keep the original, non-ASCII character in the string. Note that if 'preserve' is used the string returned by unidecode() will not be ASCII-encodable!:

>>> unidecode('\ue000') # unidecode does not have replacements for Private Use Area characters
''
>>> unidecode('\ue000', errors='strict')
Traceback (most recent call last):
...
unidecode.UnidecodeError: no replacement found for character '\ue000' in position 0

A utility is also included that allows you to transliterate text from the command line in several ways. Reading from standard input:

$ echo hello | unidecode
hello

from a command line argument:

$ unidecode -c hello
hello

or from a file:

$ unidecode hello.txt
hello

The default encoding used by the utility depends on your system locale. You can specify another encoding with the -e argument. See unidecode --help for a full list of available options.

Requirements

Nothing except Python itself. Unidecode supports Python 3.5 or later.

You need a Python build with “wide” Unicode characters (also called “UCS-4 build”) in order for Unidecode to work correctly with characters outside of Basic Multilingual Plane (BMP). Common characters outside BMP are bold, italic, script, etc. variants of the Latin alphabet intended for mathematical notation. Surrogate pair encoding of “narrow” builds is not supported in Unidecode.

If your Python build supports “wide” Unicode the following expression will return True:

>>> import sys
>>> sys.maxunicode > 0xffff
True

See PEP 261 for details regarding support for “wide” Unicode characters in Python.

Installation

To install the latest version of Unidecode from the Python package index, use these commands:

$ pip install unidecode

To install Unidecode from the source distribution and run unit tests, use:

$ python setup.py install
$ python setup.py test

Frequently asked questions

German umlauts are transliterated incorrectly

Latin letters “a”, “o” and “u” with diaeresis are transliterated by Unidecode as “a”, “o”, “u”, not according to German rules “ae”, “oe”, “ue”. This is intentional and will not be changed. Rationale is that these letters are used in languages other than German (for example, Finnish and Turkish). German text transliterated without the extra “e” is much more readable than other languages transliterated using German rules. A workaround is to do your own replacements of these characters before passing the string to unidecode().

Japanese Kanji is transliterated as Chinese

Same as with Latin letters with accents discussed in the answer above, the Unicode standard encodes letters, not letters in a certain language or their meaning. With Japanese and Chinese this is even more evident because the same letter can have very different transliterations depending on the language it is used in. Since Unidecode does not do language-specific transliteration (see next question), it must decide on one. For certain characters that are used in both Japanese and Chinese the decision was to use Chinese transliterations. If you intend to transliterate Japanese, Chinese or Korean text please consider using other libraries which do language-specific transliteration, such as Unihandecode.

Unidecode should support localization (e.g. a language or country parameter, inspecting system locale, etc.)

Language-specific transliteration is a complicated problem and beyond the scope of this library. Changes related to this will not be accepted. Please consider using other libraries which do provide this capability, such as Unihandecode.

Unidecode should automatically detect the language of the text being transliterated

Language detection is a completely separate problem and beyond the scope of this library.

Unidecode should use a permissive license such as MIT or the BSD license.

The maintainer of Unidecode believes that providing access to source code on redistribution is a fair and reasonable request when basing products on voluntary work of many contributors. If the license is not suitable for you, please consider using other libraries, such as text-unidecode.

Unidecode produces completely wrong results (e.g. “u” with diaeresis transliterating as “A 1/4 “)

The strings you are passing to Unidecode have been wrongly decoded somewhere in your program. For example, you might be decoding utf-8 encoded strings as latin1. With a misconfigured terminal, locale and/or a text editor this might not be immediately apparent. Inspect your strings with repr() and consult the Unicode HOWTO.

Why does Unidecode not replace \u and \U backslash escapes in my strings?

Unidecode knows nothing about escape sequences. Interpreting these sequences and replacing them with actual Unicode characters in string literals is the task of the Python interpreter. If you are asking this question you are very likely misunderstanding the purpose of this library. Consult the Unicode HOWTO and possibly the unicode_escape encoding in the standard library.

I’ve upgraded Unidecode and now some URLs on my website return 404 Not Found.

This is an issue with the software that is running your website, not Unidecode. Occasionally, new versions of Unidecode library are released which contain improvements to the transliteration tables. This means that you cannot rely that unidecode() output will not change across different versions of Unidecode library. If you use unidecode() to generate URLs for your website, either generate the URL slug once and store it in the database or lock your dependency of Unidecode to one specific version.

Some of the issues in this section are discussed in more detail in this blog post.

Performance notes

By default, unidecode() optimizes for the use case where most of the strings passed to it are already ASCII-only and no transliteration is necessary (this default might change in future versions).

For performance critical applications, two additional functions are exposed:

unidecode_expect_ascii() is optimized for ASCII-only inputs (approximately 5 times faster than unidecode_expect_nonascii() on 10 character strings, more on longer strings), but slightly slower for non-ASCII inputs.

unidecode_expect_nonascii() takes approximately the same amount of time on ASCII and non-ASCII inputs, but is slightly faster for non-ASCII inputs than unidecode_expect_ascii().

Apart from differences in run time, both functions produce identical results. For most users of Unidecode, the difference in performance should be negligible.

Source

You can get the latest development version of Unidecode with:

$ git clone https://www.tablix.org/~avian/git/unidecode.git

There is also an official mirror of this repository on GitHub at https://github.com/avian2/unidecode

Contact

Please make sure to read the Frequently asked questions section above before contacting the maintainer.

Bug reports, patches and suggestions for Unidecode can be sent to tomaz.solc@tablix.org.

Alternatively, you can also open a ticket or pull request at https://github.com/avian2/unidecode

Copyright

Original character transliteration tables:

Copyright 2001, Sean M. Burke <sburke@cpan.org>, all rights reserved.

Python code and later additions:

Copyright 2021, Tomaž Šolc <tomaz.solc@tablix.org>

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. The programs and documentation in this dist are distributed in the hope that they will be useful, but without any warranty; without even the implied warranty of merchantability or fitness for a particular purpose.