Skip to main content

Consolidates the shared object dependencies across multiple wheels

Project description

https://github.com/amol-/consolidatewheels/actions/workflows/tests.yml/badge.svg https://coveralls.io/repos/amol-/consolidatewheels/badge.svg https://img.shields.io/pypi/v/consolidatewheels.svg https://img.shields.io/pypi/pyversions/consolidatewheels.svg https://img.shields.io/pypi/l/consolidatewheels.svg

Supported Platforms

  • Linux

  • OSX

  • Windows

Introduction

When multiple wheels depend on each other and share some .so files, it is possible to rely on auditwheel --exclude to make sure the .so is included in only one of the wheels to avoid duplicating the file in every package.

That allows Python wheels to share the same .so as far as at least one of them is loading them in memory. See https://github.com/pypa/auditwheel/issues/76 and https://github.com/pypa/auditwheel/pull/368 for discussions on the topic.

The problem with this approach is that the package where the .so is included will have the library name mangled by auditwheel, while the ones where it’s excluded will reference to the library with its original name.

Suppose you have libone.whl and libtwo.whl both depending on libfoo.so, and libone is a dependency of libtwo so you already know you can include libfoo.so only in libone.whl, you could do:

auditwheel repair dist/libone.whl
auditwheel repair dist/libtwo.whl --exclude libfoo.so

In such case you would end up with the following wheels:

venv/lib/python3.10/site-packages/libone
├── __init__.py
├── _libone.cpython-310-aarch64-linux-gnu.so
venv/lib/python3.10/site-packages/libone.libs
└── libfoo-ef63151d.so
venv/lib/python3.10/site-packages/libtwo
├── __init__.py
├── _libtwo.cpython-310-aarch64-linux-gnu.so

The problem would be that while _libone.cpython-310-aarch64-linux-gnu.so was patched by auditwheel to know about the libfoo-ef63151d.so name, _libtwo.cpython-310-aarch64-linux-gnu.so was not, and so still refers to the original name:

$ ldd venv/lib/python3.10/site-packages/libone/_libone.cpython-310-aarch64-linux-gnu.so
    libfoo-ef63151d.so => ../libone.libs/libfoo-ef63151d.so (0x0000ffff8f8f0000)

$ ldd venv/lib/python3.10/site-packages/libtwo/_libtwo.cpython-310-aarch64-linux-gnu.so
    libfoo.so => not found

Which means that trying to import libtwo will fail with:

ImportError: libfoo.so: cannot open shared object file: No such file or directory

Which makes sense, because we actually provided libfoo-ef63151d.so and not libfoo.so.

To solve this problem consolidatewheels will patch all provided wheels to make sure that they share a single naming convention for libraries that were mangled.

After consolidatewheels is used, the final result would be:

$ ldd venv/lib/python3.10/site-packages/libone/_libone.cpython-310-aarch64-linux-gnu.so
    libfoo-ef63151d.so => ../libone.libs/libfoo-ef63151d.so (0x0000ffff8f8f0000)

$ ldd venv/lib/python3.10/site-packages/libtwo/_libtwo.cpython-310-aarch64-linux-gnu.so
    libfoo-ef63151d.so => not found

which would work correctly as far as libone is imported _before_ libtwo as they will both look for libfoo-ef63151d.so which was loaded already by libone.

Linux Support

consolidatewheels works also in conjunction with auditwheel, consolidating all libraries embedded by auditwheel. But on Linux duplicates won’t be removed automatically, so you need to make sure to use auditwheel --exclude to ensure libraries are not embedded twice.

OSX Support

consolidatewheels works also in conjunction with delocate, consolidating all libraries embedded by delocate and removing duplicates of the embedded libraries when they are provided in multiple wheels.

Windows Support

consolidatewheels works also in conjunction with delvewheel, consolidating all libraries embedded by delvewheel and removing duplicates of the embedded libraries when they are provided in multiple wheels.

Install

Install with:

$ pip install consolidatewheels

Note that consolidatewheels requires patchelf to be available in the system, and it only works on Linux systems. But those are the same requirements that auditwheel has, so you are probably already satisfying them if you use auditwheel.

Usage

Usage instructions:

consolidatewheels --help

Example:

consolidatewheels libone.whl libtwo.whl --dest=./consolidated_wheels

For a more complex example and a testing environment, you can take a look at https://github.com/amol-/wheeldeps which uses consolidatewheels

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

consolidatewheels-0.4.tar.gz (16.2 kB view details)

Uploaded Source

Built Distribution

consolidatewheels-0.4-py3-none-any.whl (14.3 kB view details)

Uploaded Python 3

File details

Details for the file consolidatewheels-0.4.tar.gz.

File metadata

  • Download URL: consolidatewheels-0.4.tar.gz
  • Upload date:
  • Size: 16.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.9.6

File hashes

Hashes for consolidatewheels-0.4.tar.gz
Algorithm Hash digest
SHA256 260cb5f4673604a5bf5386fbf7ad81f8da1832c0f1d707dc431dcd0cf4f651f1
MD5 4c9c389cfb12c7ffd326ce3e9176f60c
BLAKE2b-256 bfae74e1653ee2bd0b6c19abef37e76136481a1e8893899e966b8a6951d479a3

See more details on using hashes here.

File details

Details for the file consolidatewheels-0.4-py3-none-any.whl.

File metadata

File hashes

Hashes for consolidatewheels-0.4-py3-none-any.whl
Algorithm Hash digest
SHA256 ff8ecd2d555c83ff8bbfcaa167391a295f2cd82a9413c06a42e8eb0cf06ff19a
MD5 e9d2c599d595332abc78962f14a9bf04
BLAKE2b-256 60af99e6fd93b9f54bc7ba700479e9dc4e067e19b8b46cf9a188dadc10ac2fa0

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page