Deploy Dask on Marathon
Project description
Deploy dask-worker processes on Marathon in response to load on a Dask scheduler. This creates a Marathon application of dask-worker processes. It watches a Dask Scheduler object in the local process and, based on current requested load, scales the Marathon application up and down.
Run
It’s not yet clear how to expose all of the necessary options to a command line interface. For now we’re doing everything manually.
Make an IOLoop running in a separate thread:
with MarathonCluster(marathon='http://localhost:8080',
cpus=1, mem=512, adaptive=True) as mc:
with Client(mc.scheduler_address) as c:
x = c.submit(lambda x: x + 1, 1)
assert x.result() == 2
Create a Client and submit work to the scheduler. Marathon will scale workers up and down as neccessary in response to current workload.
from distributed import Client
c = Client(s.address)
future = c.submit(lambda x: x + 1, 10)
TODO
[x] Deploy the scheduler on the cluster
[x] Support a command line interface
Docker Testing Harness
This sets up a docker cluster of one Mesos master and two Mesos agents using docker-compose.
Requires:
docker version >= 1.11.1
docker-compose version >= 1.7.1
docker-compose up
Run py.test:
py.test dask-marathon
Web UIs
http://localhost:5050/ for Mesos master UI
http://localhost:5051/ for the first Mesos agent UI
http://localhost:8080/ for Marathon UI
History
Dask-marathon originally forked from https://github.com/mrocklin/dask-marathon
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
File details
Details for the file daskathon-1.3.1.tar.gz
.
File metadata
- Download URL: daskathon-1.3.1.tar.gz
- Upload date:
- Size: 23.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1dfb096965e61337d0825ba578c787acd50022de5e96c31ee0a4cf4a82835f70 |
|
MD5 | 9f81d543d74e00aa7525dbbb04bd4c30 |
|
BLAKE2b-256 | 793ede77ae23ae88d4ab03c5f0575316987d3e758a824d914c2a12bc1805c22a |