Skip to main content

Extract the top-level domain (TLD) from the URL given.

Project description

Extract the top level domain (TLD) from the URL given. List of TLD names is taken from Mozilla.

Optionally raises exceptions on non-existing TLDs or silently fails (if fail_silently argument is set to True).

PyPI Version Supported Python versions Build Status MPL-1.1 OR GPL-2.0-only OR LGPL-2.1-or-later Coverage

Prerequisites

  • Python 3.6, 3.7 and 3.8

Support for Python 2.7 and 3.5 is available as well.

Documentation

Documentation is available on Read the Docs.

Installation

Latest stable version on PyPI:

pip install tld

Or latest stable version from GitHub:

pip install https://github.com/barseghyanartur/tld/archive/stable.tar.gz

Or latest stable version from BitBucket:

pip install https://bitbucket.org/barseghyanartur/tld/get/stable.tar.gz

Usage examples

In addition to examples below, see the jupyter notebook workbook file.

Get the TLD name as string from the URL given

from tld import get_tld

get_tld("http://www.google.co.uk")
# 'co.uk'

get_tld("http://www.google.idontexist", fail_silently=True)
# None

Get the TLD as an object

from tld import get_tld

res = get_tld("http://some.subdomain.google.co.uk", as_object=True)

res
# 'co.uk'

res.subdomain
# 'some.subdomain'

res.domain
# 'google'

res.tld
# 'co.uk'

res.fld
# 'google.co.uk'

res.parsed_url
# SplitResult(
#     scheme='http',
#     netloc='some.subdomain.google.co.uk',
#     path='',
#     query='',
#     fragment=''
# )

Get TLD name, ignoring the missing protocol

from tld import get_tld, get_fld

get_tld("www.google.co.uk", fix_protocol=True)
# 'co.uk'

get_fld("www.google.co.uk", fix_protocol=True)
# 'google.co.uk'

Return TLD parts as tuple

from tld import parse_tld

parse_tld('http://www.google.com')
# 'com', 'google', 'www'

Get the first level domain name as string from the URL given

from tld import get_fld

get_fld("http://www.google.co.uk")
# 'google.co.uk'

get_fld("http://www.google.idontexist", fail_silently=True)
# None

Check if some tld is a valid tld

from tld import is_tld

is_tld('co.uk)
# True

is_tld('uk')
# True

is_tld('tld.doesnotexist')
# False

is_tld('www.google.com')
# False

Update the list of TLD names

To update/sync the tld names with the most recent versions run the following from your terminal:

update-tld-names

Or simply do:

from tld.utils import update_tld_names

update_tld_names()

Note, that this will update all registered TLD source parsers (not only the list of TLD names taken from Mozilla). In order to run the update for a single parser, append uid of that parser as argument.

update-tld-names mozilla

Custom TLD parsers

By default list of TLD names is taken from Mozilla. Parsing implemented in the tld.utils.MozillaTLDSourceParser class. If you want to use another parser, subclass the tld.base.BaseTLDSourceParser, provide uid, source_url, local_path and implement the get_tld_names method. Take the tld.utils.MozillaTLDSourceParser as a good example of such implementation. You could then use get_tld (as well as other tld module functions) as shown below:

from tld import get_tld
from some.module import CustomTLDSourceParser

get_tld(
    "http://www.google.co.uk",
    parser_class=CustomTLDSourceParser
)

Custom list of TLD names

You could maintain your own custom version of the TLD names list (even multiple ones) and use them simultaneously with built in TLD names list.

You would then store them locally and provide a path to it as shown below:

from tld import get_tld
from tld.utils import BaseMozillaTLDSourceParser

class CustomBaseMozillaTLDSourceParser(BaseMozillaTLDSourceParser):

    uid: str = 'custom_mozilla'
    local_path: str = 'tests/res/effective_tld_names_custom.dat.txt'

get_tld(
    "http://www.foreverchild",
    parser_class=CustomBaseMozillaTLDSourceParser
)
# 'foreverchild'

Same goes for first level domain names:

from tld import get_fld

get_fld(
    "http://www.foreverchild",
    parser_class=CustomBaseMozillaTLDSourceParser
)
# 'www.foreverchild'

Note, that in both examples shown above, there the original TLD names file has been modified in the following way:

...
// ===BEGIN ICANN DOMAINS===

// This one actually does not exist, added for testing purposes
foreverchild
...

Free up resources

To free up memory occupied by loading of custom TLD names, use reset_tld_names function with tld_names_local_path parameter.

from tld import get_tld, reset_tld_names

# Get TLD from a custom TLD names parser
get_tld(
    "http://www.foreverchild",
    parser_class=CustomBaseMozillaTLDSourceParser
)

# Free resources occupied by the custom TLD names list
reset_tld_names("tests/res/effective_tld_names_custom.dat.txt")

Support for Python 2.7 and 3.5

As you might have noticed, typing (Python 3.6+) is extensively used in the code. However, Python 3.5 will likely be supported until it’s EOL. All modern recent versions (starting from tld``==0.11.7) are fully compatible with Python 2.7 and 3.5 (just works with ``pip install tld).

Development tips follow:

Python 2.7

Install from pip

pip install tld[py27]

Install locally in development mode

python setup.py develop --python-tag py27

Prepare dist

./scripts/prepare_build_py27.sh

Run tests

./runtests.py src_py27/ --python-tag py27

Python 3.5

Install from pip

pip install tld[py35]

Install locally in development mode

python setup.py develop --python-tag py35

Prepare dist

./scripts/prepare_build_py35.sh

Run tests

./runtests.py src_py35/ --python-tag py35

Troubleshooting

If somehow domain names listed here are not recognised, make sure you have the most recent version of TLD names in your virtual environment:

update-tld-names

To update TLD names list for a single parser, specify it as an argument:

update-tld-names mozilla

Testing

Simply type:

./runtests.py

Or use tox:

tox

Or use tox to check specific env:

tox -e py36

Writing documentation

Keep the following hierarchy.

=====
title
=====

header
======

sub-header
----------

sub-sub-header
~~~~~~~~~~~~~~

sub-sub-sub-header
^^^^^^^^^^^^^^^^^^

sub-sub-sub-sub-header
++++++++++++++++++++++

sub-sub-sub-sub-sub-header
**************************

License

MPL-1.1 OR GPL-2.0-only OR LGPL-2.1-or-later

Support

For any issues contact me at the e-mail given in the Author section.

Author

Artur Barseghyan <artur.barseghyan@gmail.com>

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tld-0.11.8.tar.gz (677.8 kB view details)

Uploaded Source

Built Distributions

tld-0.11.8-py38-none-any.whl (258.5 kB view details)

Uploaded Python 3.8

tld-0.11.8-py37-none-any.whl (258.5 kB view details)

Uploaded Python 3.7

tld-0.11.8-py36-none-any.whl (258.5 kB view details)

Uploaded Python 3.6

tld-0.11.8-py35-none-any.whl (258.1 kB view details)

Uploaded Python 3.5

tld-0.11.8-py27-none-any.whl (258.3 kB view details)

Uploaded Python 2.7

File details

Details for the file tld-0.11.8.tar.gz.

File metadata

  • Download URL: tld-0.11.8.tar.gz
  • Upload date:
  • Size: 677.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.14.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.35.0 CPython/3.6.9

File hashes

Hashes for tld-0.11.8.tar.gz
Algorithm Hash digest
SHA256 05db1b7e59cd937424a18e315b8b660933d6dfbae35517e736e566d45858a8e8
MD5 0faef18ada7df6cef5efc0e3194b7be0
BLAKE2b-256 658bed33e02490bca1de7e996d37f4e05887cc2cb7a50606c12f6e0663d6df3f

See more details on using hashes here.

Provenance

File details

Details for the file tld-0.11.8-py38-none-any.whl.

File metadata

  • Download URL: tld-0.11.8-py38-none-any.whl
  • Upload date:
  • Size: 258.5 kB
  • Tags: Python 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.14.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.35.0 CPython/3.6.9

File hashes

Hashes for tld-0.11.8-py38-none-any.whl
Algorithm Hash digest
SHA256 0d87e732d1974c333739e75902f43b42db093fc7110b6be51c8521f25838bd15
MD5 eab3496b5ce36b75eda2c8b64b2368d8
BLAKE2b-256 884d0bd05c97b4a1303c2d788746d628e94470d5cdcf1559cb5dc950df62ed67

See more details on using hashes here.

Provenance

File details

Details for the file tld-0.11.8-py37-none-any.whl.

File metadata

  • Download URL: tld-0.11.8-py37-none-any.whl
  • Upload date:
  • Size: 258.5 kB
  • Tags: Python 3.7
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.14.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.35.0 CPython/3.6.9

File hashes

Hashes for tld-0.11.8-py37-none-any.whl
Algorithm Hash digest
SHA256 e7bb3d4384cbac378d6a5652402067a140ee64a008f6a9c268a0decd11c2b607
MD5 dc7a85697e6eac28a3cb75892dc7bd39
BLAKE2b-256 0538a8d4edf4bb25a2474045a1b3f1cd833a42981d982993d830ccdaaa65c1f0

See more details on using hashes here.

Provenance

File details

Details for the file tld-0.11.8-py36-none-any.whl.

File metadata

  • Download URL: tld-0.11.8-py36-none-any.whl
  • Upload date:
  • Size: 258.5 kB
  • Tags: Python 3.6
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.14.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.35.0 CPython/3.6.9

File hashes

Hashes for tld-0.11.8-py36-none-any.whl
Algorithm Hash digest
SHA256 ae12633ca0815c1abdccbaa86688135571b46c4bbea73482188d667f1cffbe90
MD5 d67609007d0cd57a854b44920a6b2928
BLAKE2b-256 f601fa97eff7a31462eec88d642ac01f9957c25dd15239df5f36c56ecdecbdac

See more details on using hashes here.

Provenance

File details

Details for the file tld-0.11.8-py35-none-any.whl.

File metadata

  • Download URL: tld-0.11.8-py35-none-any.whl
  • Upload date:
  • Size: 258.1 kB
  • Tags: Python 3.5
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.14.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.35.0 CPython/3.6.9

File hashes

Hashes for tld-0.11.8-py35-none-any.whl
Algorithm Hash digest
SHA256 6bc033da28e30281e735cc444a94b20ad8ca3f91af54e13450bf7dae6af5f892
MD5 1ffb56e2d4e3e206de60fc0acc152744
BLAKE2b-256 1e05cfb3c78245b50bde283f638f56479bb151338514946e7350e188f920a56b

See more details on using hashes here.

Provenance

File details

Details for the file tld-0.11.8-py27-none-any.whl.

File metadata

  • Download URL: tld-0.11.8-py27-none-any.whl
  • Upload date:
  • Size: 258.3 kB
  • Tags: Python 2.7
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.14.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.35.0 CPython/3.6.9

File hashes

Hashes for tld-0.11.8-py27-none-any.whl
Algorithm Hash digest
SHA256 54cf98715a1af0792a788ba96dc8c18a3a3034c8a4b6a5917cae81c2d6ee9a52
MD5 801c36e4d693e2ec8abb9a321016f75b
BLAKE2b-256 7618ea2ae873f9978a6a4c7fbfe11f17e1052d271f6b5544a4669a51137d2e45

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page