python-libguess 1.0

determine_encoding(in_string, region)

Determines encoding of a string for the given language region.

As an usage example we might see what happens when given Japanese text in 2 different encodings:

>>> import guess
>>> guess.determine_encoding(u'\u3042'.encode('euc-jp'), guess.REGION_JP)
'EUC-JP'
>>> guess.determine_encoding(u'\u3042'.encode('utf-8'), guess.REGION_JP)
'UTF-8'

The output string of this function can be given directly to iconv_open() C function and the resulting names should be compatible with the encoding string of str.decode() function:

>>> encoded_value = u'\u3042'.encode('shift-jis')
>>> encoding = guess.determine_encoding(encoded_value, guess.REGION_JP)
>>> encoding
'SJIS'
>>> encoded_value.decode(encoding)
u'\u3042'

In case the given region name is invalid or the underlying libguess_determine_encoding() call fails for any other reason, None value is returned:

>>> encoding = guess.determine_encoding("asdf", "UNKNOWN")
>>> encoding is None
True

Use REGION_* constants for region names.

validate_utf8(in_string)

Checks if the given string is a valid UTF-8 byte sequence.

This function is included here for the completeness with libguess interface. It gives more precise results for UTF-8 validity than for example functions in glib.

>>> import guess
>>> guess.validate_utf8(u'\u3042'.encode('EUC-JP'))
False
>>> guess.validate_utf8(u'\u3042'.encode('UTF-8'))
True