Skip to main content

Launchpad is a library that simplifies writing distributed programs and seamlessly launching them on a range of supported platforms.

Project description

Launchpad

PyPI - Python Version PyPI version

Launchpad is a library that simplifies writing distributed programs by seamlessly launching them on a variety of different platforms. Switching between local and distributed execution requires only a flag change.

Launchpad introduces a programming model that represents a distributed system as a graph data structure (a Program) describing the system’s topology. Each node in the program graph represents a service in the distributed system, i.e. the fundamental unit of computation that we are interested in running. As nodes are added to this graph, Launchpad constructs a handle for each of them. A handle ultimately represents a client to the yet-to-be-constructed service. A directed edge in the program graph, representing communication between two services, is created when the handle associated with one node is given to another at construction time. This edge originates from the receiving node, indicating that the receiving node will be the one initiating communication. This process allows Launchpad to define cross-service communication simply by passing handles to nodes. Launchpad provides a number of node types, including:

  • PyNode - a simple node executing provided Python code upon entry. It is similar to a main function, but with the distinction that each node may be running in separate processes and on different machines.
  • CourierNode - it enables cross-node communication. CourierNodes can communicate by calling public methods on each other either synchronously or asynchronously via futures. The underlying remote procedure calls are handled transparently by Launchpad.
  • ReverbNode - it exposes functionality of Reverb, an easy-to-use data storage and transport system primarily used by RL algorithms as an experience replay. You can read more about Reverb here.
  • MultiThreadingColocation - allows to colocate multiple other nodes in a single process.
  • MultiProcessingColocation - allows to colocate multiple other nodes as sub processes.

Using Launchpad involves writing nodes and defining the topology of your distributed program by passing to each node references of the other nodes that it can communicate with. The core data structure dealing with this is called a Launchpad program, which can then be executed seamlessly with a number of supported runtimes.

Supported launch types

Launchpad supports a number of launch types, both for running programs on a single machine, in a distributed manner, or in a form of a test. Launch type can be controlled by the launch_type argument passed to lp.launch method, or specified through the --lp_launch_type command line flag. Please refer to the documentation of the LaunchType for details.

Table of Contents

Installation

Please keep in mind that Launchpad is not hardened for production use, and while we do our best to keep things in working order, things may break or segfault.

:warning: Launchpad currently only supports Linux based OSes.

The recommended way to install Launchpad is with pip. We also provide instructions to build from source using the same docker images we use for releases.

TensorFlow can be installed separately or as part of the pip install. Installing TensorFlow as part of the install ensures compatibility.

$ pip install dm-launchpad[tensorflow]

# Without Tensorflow install and version dependency check.
$ pip install dm-launchpad

Nightly builds

PyPI version

$ pip install dm-launchpad-nightly[tensorflow]

# Without Tensorflow install and version dependency check.
$ pip install dm-launchpad-nightly

Similarily, Reverb can be installed ensuring compatibility:

$ pip install dm-launchpad[reverb]

Develop Launchpad inside a docker container

The most convenient way to develop Launchpad is with Docker. This way you can compile and test Launchpad inside a container without having to install anything on your host machine, while you can still use your editor of choice for making code changes. The steps are as follows.

Checkout Launchpad's source code from GitHub.

$ git checkout https://github.com/deepmind/launchpad.git
$ cd launchpad

Build the Docker container to be used for compiling and testing Launchpad. You can specify tensorflow_pip parameter to set the version of Tensorflow to build against. You can also specify which version(s) of Python container should support. The command below enables support for Python 3.7, 3.8, 3.9 and 3.10.

$ docker build --tag launchpad:devel \
  --build-arg tensorflow_pip=tensorflow==2.3.0 \
  --build-arg python_version="3.7 3.8 3.9 3.10" - < docker/build.dockerfile

The next step is to enter the built Docker image, binding checked out Launchpad's sources to /tmp/launchpad within the container.

$ docker run --rm --mount "type=bind,src=$PWD,dst=/tmp/launchpad" \
  -it launchpad:devel bash

At this point you can build and install Launchpad within the container by executing:

$ /tmp/launchpad/oss_build.sh

By default it builds Python 3.8 version, you can change that with --python flag.

$ /tmp/launchpad/oss_build.sh --python 3.8

To make sure installation was successful and Launchpad works as expected, you can run some examples provided:

$ python3.8 -m launchpad.examples.hello_world.launch
$ python3.8 -m launchpad.examples.consumer_producers.launch --lp_launch_type=local_mp

To make changes to Launchpad codebase, edit sources checked out from GitHub directly on your host machine (outside of the Docker container). All changes are visible inside the Docker container. To recompile just run the oss_build.sh script again from the Docker container. In order to reduce compilation time of the consecutive runs, make sure to not exit the Docker container.

Citing Launchpad

If you use Launchpad in your work, please cite the accompanying technical report:

@article{yang2021launchpad,
    title={Launchpad: A Programming Model for Distributed Machine Learning
           Research},
    author={Fan Yang and Gabriel Barth-Maron and Piotr Stańczyk and Matthew
            Hoffman and Siqi Liu and Manuel Kroiss and Aedan Pope and Alban
            Rrustemi},
    year={2021},
    journal={arXiv preprint arXiv:2106.04516},
    url={https://arxiv.org/abs/2106.04516},
}

Acknowledgements

We greatly appreciate all the help from Reverb and TF-Agents teams in setting up building and testing setup for Launchpad.

Other resources

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

dm_launchpad_nightly-0.3.0.dev20220308-cp310-cp310-manylinux2010_x86_64.whl (4.0 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.12+ x86-64

dm_launchpad_nightly-0.3.0.dev20220308-cp39-cp39-manylinux2010_x86_64.whl (4.0 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.12+ x86-64

dm_launchpad_nightly-0.3.0.dev20220308-cp38-cp38-manylinux2010_x86_64.whl (4.0 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.12+ x86-64

dm_launchpad_nightly-0.3.0.dev20220308-cp37-cp37m-manylinux2010_x86_64.whl (4.0 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.12+ x86-64

File details

Details for the file dm_launchpad_nightly-0.3.0.dev20220308-cp310-cp310-manylinux2010_x86_64.whl.

File metadata

  • Download URL: dm_launchpad_nightly-0.3.0.dev20220308-cp310-cp310-manylinux2010_x86_64.whl
  • Upload date:
  • Size: 4.0 MB
  • Tags: CPython 3.10, manylinux: glibc 2.12+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/33.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.63.0 importlib-metadata/4.8.3 keyring/23.4.1 rfc3986/1.5.0 colorama/0.4.4 CPython/3.6.9

File hashes

Hashes for dm_launchpad_nightly-0.3.0.dev20220308-cp310-cp310-manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 59e2048c8928a772f68b9206299e1d382f5f547fe2fccad9cfc569915fe240c4
MD5 61d473e0effe0a3cd5d90d0f6cf75b5c
BLAKE2b-256 570c1429937d4590dcb04300a276b84881fbe4fdd6986ff8c4a98851084fff46

See more details on using hashes here.

File details

Details for the file dm_launchpad_nightly-0.3.0.dev20220308-cp39-cp39-manylinux2010_x86_64.whl.

File metadata

  • Download URL: dm_launchpad_nightly-0.3.0.dev20220308-cp39-cp39-manylinux2010_x86_64.whl
  • Upload date:
  • Size: 4.0 MB
  • Tags: CPython 3.9, manylinux: glibc 2.12+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/33.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.63.0 importlib-metadata/4.8.3 keyring/23.4.1 rfc3986/1.5.0 colorama/0.4.4 CPython/3.6.9

File hashes

Hashes for dm_launchpad_nightly-0.3.0.dev20220308-cp39-cp39-manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 abb30668326dd268cdc5ff1d80da5680e28fa5ea76c926a2275209e1e57acd1a
MD5 0b506fafddb5c1144c3d8daa349595e3
BLAKE2b-256 5c9866d8ff40c3491f2e667b9dbf6776a2e2460a1a8e01432ad2ed060f49cdd9

See more details on using hashes here.

File details

Details for the file dm_launchpad_nightly-0.3.0.dev20220308-cp38-cp38-manylinux2010_x86_64.whl.

File metadata

  • Download URL: dm_launchpad_nightly-0.3.0.dev20220308-cp38-cp38-manylinux2010_x86_64.whl
  • Upload date:
  • Size: 4.0 MB
  • Tags: CPython 3.8, manylinux: glibc 2.12+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/33.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.63.0 importlib-metadata/4.8.3 keyring/23.4.1 rfc3986/1.5.0 colorama/0.4.4 CPython/3.6.9

File hashes

Hashes for dm_launchpad_nightly-0.3.0.dev20220308-cp38-cp38-manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 0cd85bc154884f9031f5122982bd23a0c892e6e0d7a126d1356685f46b383950
MD5 39bbc8bbcdad0f039139ecb9459ca8a1
BLAKE2b-256 71f5adcc438f9ba34c0d1562077d8973aaac773e636931fd5be6f2b5d77f4a88

See more details on using hashes here.

File details

Details for the file dm_launchpad_nightly-0.3.0.dev20220308-cp37-cp37m-manylinux2010_x86_64.whl.

File metadata

  • Download URL: dm_launchpad_nightly-0.3.0.dev20220308-cp37-cp37m-manylinux2010_x86_64.whl
  • Upload date:
  • Size: 4.0 MB
  • Tags: CPython 3.7m, manylinux: glibc 2.12+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/33.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.63.0 importlib-metadata/4.8.3 keyring/23.4.1 rfc3986/1.5.0 colorama/0.4.4 CPython/3.6.9

File hashes

Hashes for dm_launchpad_nightly-0.3.0.dev20220308-cp37-cp37m-manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 b62b18b3ff8c2d9fdaaf5c2205247aa373ebdf3c22fdc7605e2916e5a081fcf2
MD5 50148c8b558babb412d55b7e0b15928e
BLAKE2b-256 1a7aa47ad42a9277f68b733e4ccd7e1d1e552b3c60fdcf14e82627c8c057d2c8

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page