yarl 1.10.0

Introduction

Url is constructed from str:

>>> from yarl import URL
>>> url = URL('https://www.python.org/~guido?arg=1#frag')
>>> url
URL('https://www.python.org/~guido?arg=1#frag')

All url parts: scheme, user, password, host, port, path, query and fragment are accessible by properties:

>>> url.scheme
'https'
>>> url.host
'www.python.org'
>>> url.path
'/~guido'
>>> url.query_string
'arg=1'
>>> url.query
<MultiDictProxy('arg': '1')>
>>> url.fragment
'frag'

All url manipulations produce a new url object:

>>> url = URL('https://www.python.org')
>>> url / 'foo' / 'bar'
URL('https://www.python.org/foo/bar')
>>> url / 'foo' % {'bar': 'baz'}
URL('https://www.python.org/foo?bar=baz')

Strings passed to constructor and modification methods are automatically encoded giving canonical representation as result:

>>> url = URL('https://www.python.org/шлях')
>>> url
URL('https://www.python.org/%D1%88%D0%BB%D1%8F%D1%85')

Regular properties are percent-decoded, use raw_ versions for getting encoded strings:

>>> url.path
'/шлях'

>>> url.raw_path
'/%D1%88%D0%BB%D1%8F%D1%85'

Human readable representation of URL is available as .human_repr():

>>> url.human_repr()
'https://www.python.org/шлях'

For full documentation please read https://yarl.aio-libs.org.

Installation

$ pip install yarl

The library is Python 3 only!

PyPI contains binary wheels for Linux, Windows and MacOS. If you want to install yarl on another operating system (like Alpine Linux, which is not manylinux-compliant because of the missing glibc and therefore, cannot be used with our wheels) the the tarball will be used to compile the library from the source code. It requires a C compiler and and Python headers installed.

To skip the compilation you must explicitly opt-in by using a PEP 517 configuration setting pure-python, or setting the YARL_NO_EXTENSIONS environment variable to a non-empty value, e.g.:

$ pip install yarl --config-settings=pure-python=false

Please note that the pure-Python (uncompiled) version is much slower. However, PyPy always uses a pure-Python implementation, and, as such, it is unaffected by this variable.

Dependencies

YARL requires multidict library.

API documentation

The documentation is located at https://yarl.aio-libs.org.

Why isn’t boolean supported by the URL query API?

There is no standard for boolean representation of boolean values.

Some systems prefer true/false, others like yes/no, on/off, Y/N, 1/0, etc.

yarl cannot make an unambiguous decision on how to serialize bool values because it is specific to how the end-user’s application is built and would be different for different apps. The library doesn’t accept booleans in the API; a user should convert bools into strings using own preferred translation protocol.

Comparison with other URL libraries

• furl (https://pypi-hypernode.com/pypi/furl)

  The library has rich functionality but the furl object is mutable.

  I’m afraid to pass this object into foreign code: who knows if the code will modify my url in a terrible way while I just want to send URL with handy helpers for accessing URL properties.

  furl has other non-obvious tricky things but the main objection is mutability.

• URLObject (https://pypi-hypernode.com/pypi/URLObject)

  URLObject is immutable, that’s pretty good.

  Every URL change generates a new URL object.

  But the library doesn’t do any decode/encode transformations leaving the end user to cope with these gory details.

Source code

The project is hosted on GitHub

Please file an issue on the bug tracker if you have found a bug or have some suggestion in order to improve the library.

Discussion list

aio-libs google group: https://groups.google.com/forum/#!forum/aio-libs

Feel free to post your questions and ideas here.

Authors and License

The yarl package is written by Andrew Svetlov.

It’s Apache 2 licensed and freely available.

Bug fixes

• Fixed joining a path when the existing path was empty – by @bdraco.

  A regression in URL.join()() was introduced in #1082.

  Related issues and pull requests on GitHub: #1118.

Features

• Added URL.without_query_params()() method, to drop some parameters from query string – by @hongquan.

  Related issues and pull requests on GitHub: #774, #898, #1010.

• The previously protected types _SimpleQuery, _QueryVariable, and _Query are now available for use externally as SimpleQuery, QueryVariable, and Query – by @bdraco.

  Related issues and pull requests on GitHub: #1050, #1113.

Contributor-facing changes

• Replaced all ~typing.Optional with ~typing.Union – by @bdraco.

  Related issues and pull requests on GitHub: #1095.

Miscellaneous internal changes

• Significantly improved performance of parsing the network location – by @bdraco.

  Related issues and pull requests on GitHub: #1112.

• Added internal types to the cache to prevent future refactoring errors – by @bdraco.

  Related issues and pull requests on GitHub: #1117.

Bug fixes

• Fixed a TypeError with MultiDictProxy and Python 3.8 – by @bdraco.

  Related issues and pull requests on GitHub: #1084, #1105, #1107.

Miscellaneous internal changes

• Improved performance of encoding hosts – by @bdraco.

  Previously, the library would unconditionally try to parse a host as an IP Address. The library now avoids trying to parse a host as an IP Address if the string is not in one of the formats described in 3986#section-3.2.2.

  Related issues and pull requests on GitHub: #1104.

Bug fixes

• URL.join()() has been changed to match 3986 and align with / operation() and URL.joinpath()() when joining URLs with empty segments. Previously urllib.parse.urljoin was used, which has known issues with empty segments (python/cpython#84774).

  Due to the semantics of URL.join()(), joining an URL with scheme requires making it relative, prefixing with ./.

  >>> URL("https://web.archive.org/web/").join(URL("./https://github.com/aio-libs/yarl"))
  URL('https://web.archive.org/web/https://github.com/aio-libs/yarl')

  Empty segments are honored in the base as well as the joined part.

  >>> URL("https://web.archive.org/web/https://").join(URL("github.com/aio-libs/yarl"))
  URL('https://web.archive.org/web/https://github.com/aio-libs/yarl')

  – by @commonism

  This change initially appeared in 1.9.5 but was reverted in 1.9.6 to resolve a problem with query string handling.

  Related issues and pull requests on GitHub: #1039, #1082.

Features

• Added ~yarl.URL.absolute which is now preferred over URL.is_absolute() – by @bdraco.

  Related issues and pull requests on GitHub: #1100.

Bug fixes

• Added missing type on ~yarl.URL.port – by @bdraco.

  Related issues and pull requests on GitHub: #1097.

Features

• Covered the ~yarl.URL object with types – by @bdraco.

  Related issues and pull requests on GitHub: #1084.

• Cache parsing of IP Addresses when encoding hosts – by @bdraco.

  Related issues and pull requests on GitHub: #1086.

Contributor-facing changes

• Covered the ~yarl.URL object with types – by @bdraco.

  Related issues and pull requests on GitHub: #1084.

Miscellaneous internal changes

• Improved performance of handling ports – by @bdraco.

  Related issues and pull requests on GitHub: #1081.

Removals and backward incompatible breaking changes

• Removed support 3986#section-3.2.3 port normalization when the scheme is not one of http, https, wss, or ws – by @bdraco.

  Support for port normalization was recently added in #1033 and contained code that would do blocking I/O if the scheme was not one of the four listed above. The code has been removed because this library is intended to be safe for usage with asyncio.

  Related issues and pull requests on GitHub: #1076.

Miscellaneous internal changes

• Improved performance of property caching – by @bdraco.

  The reify implementation from aiohttp was adapted to replace the internal cached_property implementation.

  Related issues and pull requests on GitHub: #1070.

Bug fixes

• Reverted 3986 compatible URL.join()() honoring empty segments which was introduced in #1039.

  This change introduced a regression handling query string parameters with joined URLs. The change was reverted to maintain compatibility with the previous behavior.

  Related issues and pull requests on GitHub: #1067.

Bug fixes

• Joining URLs with empty segments has been changed to match 3986.

  Previously empty segments would be removed from path, breaking use-cases such as

  URL("https://web.archive.org/web/") / "https://github.com/"

  Now / operation() and URL.joinpath()() keep empty segments, but do not introduce new empty segments. e.g.

  URL("https://example.org/") / ""

  does not introduce an empty segment.

  – by @commonism and @youtux

  Related issues and pull requests on GitHub: #1026.

• The default protocol ports of well-known URI schemes are now taken into account during the normalization of the URL string representation in accordance with 3986#section-3.2.3.

  Specified ports are removed from the str representation of a ~yarl.URL if the port matches the scheme’s default port – by @commonism.

  Related issues and pull requests on GitHub: #1033.

• URL.join()() has been changed to match 3986 and align with / operation() and URL.joinpath()() when joining URLs with empty segments. Previously urllib.parse.urljoin was used, which has known issues with empty segments (python/cpython#84774).

  Due to the semantics of URL.join()(), joining an URL with scheme requires making it relative, prefixing with ./.

  >>> URL("https://web.archive.org/web/").join(URL("./https://github.com/aio-libs/yarl"))
  URL('https://web.archive.org/web/https://github.com/aio-libs/yarl')

  Empty segments are honored in the base as well as the joined part.

  >>> URL("https://web.archive.org/web/https://").join(URL("github.com/aio-libs/yarl"))
  URL('https://web.archive.org/web/https://github.com/aio-libs/yarl')

  – by @commonism

  Related issues and pull requests on GitHub: #1039.

Removals and backward incompatible breaking changes

• Stopped decoding %2F (/) in URL.path, as this could lead to code incorrectly treating it as a path separator – by @Dreamsorcerer.

  Related issues and pull requests on GitHub: #1057.

• Dropped support for Python 3.7 – by @Dreamsorcerer.

  Related issues and pull requests on GitHub: #1016.

Improved documentation

• On the Contributing docs page, a link to the Towncrier philosophy has been fixed.

  Related issues and pull requests on GitHub: #981.

• The pre-existing / magic method() has been documented in the API reference – by @commonism.

  Related issues and pull requests on GitHub: #1026.

Packaging updates and notes for downstreams

• A flaw in the logic for copying the project directory into a temporary folder that led to infinite recursion when TMPDIR was set to a project subdirectory path. This was happening in Fedora and its downstream due to the use of pyproject-rpm-macros. It was only reproducible with pip wheel and was not affecting the pyproject-build users.

  – by @hroncok and @webknjaz

  Related issues and pull requests on GitHub: #992, #1014.

• Support Python 3.13 and publish non-free-threaded wheels

  Related issues and pull requests on GitHub: #1054.

Contributor-facing changes

• The CI/CD setup has been updated to test arm64 wheels under macOS 14, except for Python 3.7 that is unsupported in that environment – by @webknjaz.

  Related issues and pull requests on GitHub: #1015.

• Removed unused type ignores and casts – by @hauntsaninja.

  Related issues and pull requests on GitHub: #1031.

Miscellaneous internal changes

• port, scheme, and raw_host are now cached_property – by @bdraco.

  aiohttp accesses these properties quite often, which cause urllib to build the _hostinfo property every time. port, scheme, and raw_host are now cached properties, which will improve performance.

  Related issues and pull requests on GitHub: #1044, #1058.

Bug fixes

• Started raising TypeError when a string value is passed into yarl.URL.build() as the port argument – by @commonism.

  Previously the empty string as port would create malformed URLs when rendered as string representations. (#883)

Packaging updates and notes for downstreams

• The leading -- has been dropped from the PEP 517 in-tree build backend config setting names. --pure-python is now just pure-python – by @webknjaz.

  The usage now looks as follows:

  $ python -m build \
      --config-setting=pure-python=true \
      --config-setting=with-cython-tracing=true

  (#963)

Contributor-facing changes

• A step-by-step Release Guide guide has been added, describing how to release yarl – by @webknjaz.

  This is primarily targeting maintainers. (#960)

• Coverage collection has been implemented for the Cython modules – by @webknjaz.

  It will also be reported to Codecov from any non-release CI jobs.

  To measure coverage in a development environment, yarl can be installed in editable mode:

  $ python -Im pip install -e .

  Editable install produces C-files required for the Cython coverage plugin to map the measurements back to the PYX-files.

  #961

• It is now possible to request line tracing in Cython builds using the with-cython-tracing PEP 517 config setting – @webknjaz.

  This can be used in CI and development environment to measure coverage on Cython modules, but is not normally useful to the end-users or downstream packagers.

  Here’s a usage example:

  $ python -Im pip install . --config-settings=with-cython-tracing=true

  For editable installs, this setting is on by default. Otherwise, it’s off unless requested explicitly.

  The following produces C-files required for the Cython coverage plugin to map the measurements back to the PYX-files:

  $ python -Im pip install -e .

  Alternatively, the YARL_CYTHON_TRACING=1 environment variable can be set to do the same as the PEP 517 config setting.

  #962

Bug fixes

• Stopped dropping trailing slashes in yarl.URL.joinpath() – by @gmacon. (#862, #866)

• Started accepting string subclasses in yarl.URL.__truediv__() operations (URL / segment) – by @mjpieters. (#871, #884)

• Fixed the human representation of URLs with square brackets in usernames and passwords – by @mjpieters. (#876, #882)

• Updated type hints to include URL.missing_port(), URL.__bytes__() and the encoding argument to yarl.URL.joinpath() – by @mjpieters. (#891)

Packaging updates and notes for downstreams

• Integrated Cython 3 to enable building yarl under Python 3.12 – by @mjpieters. (#829, #881)

• Declared modern setuptools.build_meta as the PEP 517 build backend in pyproject.toml explicitly – by @webknjaz. (#886)

• Converted most of the packaging setup into a declarative setup.cfg config – by @webknjaz. (#890)

• The packaging is replaced from an old-fashioned setup.py to an in-tree PEP 517 build backend – by @webknjaz.

  Whenever the end-users or downstream packagers need to build yarl from source (a Git checkout or an sdist), they may pass a config_settings flag --pure-python. If this flag is not set, a C-extension will be built and included into the distribution.

  Here is how this can be done with pip:

  $ python -m pip install . --config-settings=--pure-python=false

  This will also work with -e | --editable.

  The same can be achieved via pypa/build:

  $ python -m build --config-setting=--pure-python=false

  Adding -w | --wheel can force pypa/build produce a wheel from source directly, as opposed to building an sdist and then building from it. (#893)

• Declared Python 3.12 supported officially in the distribution package metadata – by @edgarrmondragon. (#942)

Contributor-facing changes

• A regression test for no-host URLs was added per #821 and 3986 – by @kenballus. (#821, #822)

• Started testing yarl against Python 3.12 in CI – by @mjpieters. (#881)

• All Python 3.12 jobs are now marked as required to pass in CI – by @edgarrmondragon. (#942)

• MyST is now integrated in Sphinx – by @webknjaz.

  This allows the contributors to author new documents in Markdown when they have difficulties with going straight RST. (#953)

Features

• Added URL.joinpath(*elements), to create a new URL appending multiple path elements. (#704)

• Made URL.__truediv__()() return NotImplemented if called with an unsupported type — by @michaeljpeters. (#832)

Bugfixes

• Path normalization for absolute URLs no longer raises a ValueError exception when .. segments would otherwise go beyond the URL path root. (#536)

• Fixed an issue with update_query() not getting rid of the query when argument is None. (#792)

• Added some input restrictions on with_port() function to prevent invalid boolean inputs or out of valid port inputs; handled incorrect 0 port representation. (#793)

• Made yarl.URL.build() raise a TypeError if the host argument is None — by @paulpapacz. (#808)

• Fixed an issue with update_query() getting rid of the query when the argument is empty but not None. (#845)

Misc

• #220

Features

• Added URL.raw_suffix, URL.suffix, URL.raw_suffixes, URL.suffixes, URL.with_suffix. (#613)

Improved Documentation

• Fixed broken internal references to yarl.URL.human_repr(). (#665)

• Fixed broken external references to multidict:index docs. (#665)

Deprecations and Removals

• Dropped Python 3.6 support. (#672)

Misc

• #646, #699, #701

Features

• Provide wheels for aarch64, i686, ppc64le, s390x architectures on Linux as well as x86_64. #507

• Provide wheels for Python 3.9. #526

Bugfixes

• human_repr() now always produces valid representation equivalent to the original URL (if the original URL is valid). #511

• Fixed requoting a single percent followed by a percent-encoded character in the Cython implementation. #514

• Fix ValueError when decoding % which is not followed by two hexadecimal digits. #516

• Fix decoding % followed by a space and hexadecimal digit. #520

• Fix annotation of with_query()/update_query() methods for key=[val1, val2] case. #528

Removal

• Drop Python 3.5 support; Python 3.6 is the minimal supported Python version.

Features

• Allow for int and float subclasses in query, while still denying bool. #492

Bugfixes

• Do not requote arguments in URL.build(), with_xxx() and in / operator. #502

• Keep IPv6 brackets in origin(). #504

Bugfixes

• Fix including relocated internal yarl._quoting_c C-extension into published PyPI dists. #485

Misc

• #484

Features

• Convert host to lowercase on URL building. #386

• Allow using mod operator (%) for updating query string (an alias for update_query() method). #435

• Allow use of sequences such as list and tuple in the values of a mapping such as dict to represent that a key has many values:

  url = URL("http://example.com")
  assert url.with_query({"a": [1, 2]}) == URL("http://example.com/?a=1&a=2")

  #443

• Support URL.build() with scheme and path (creates a relative URL). #464

• Cache slow IDNA encode/decode calls. #476

• Add @final / Final type hints #477

• Support URL authority/raw_authority properties and authority argument of URL.build() method. #478

• Hide the library implementation details, make the exposed public list very clean. #483

Bugfixes

• Fix tests with newer Python (3.7.6, 3.8.1 and 3.9.0+). #409

• Fix a bug where query component, passed in a form of mapping or sequence, is unquoted in unexpected way. #426

• Hide Query and QueryVariable type aliases in __init__.pyi, now they are prefixed with underscore. #431

• Keep IPv6 brackets after updating port/user/password. #451