Skip to main content

Deploy Dask on Marathon

Project description

Build Status Gitential Coding Hours

Deploy dask-worker processes on Marathon in response to load on a Dask scheduler. This creates a Marathon application of dask-worker processes. It watches a Dask Scheduler object in the local process and, based on current requested load, scales the Marathon application up and down.

Run

It’s not yet clear how to expose all of the necessary options to a command line interface. For now we’re doing everything manually.

Make an IOLoop running in a separate thread:

with MarathonCluster(marathon='http://localhost:8080',
                     cpus=1, mem=512, adaptive=True) as mc:
    with Client(mc.scheduler_address) as c:
        x = c.submit(lambda x: x + 1, 1)
        assert x.result() == 2

Create a Client and submit work to the scheduler. Marathon will scale workers up and down as neccessary in response to current workload.

from distributed import Client
c = Client(s.address)

future = c.submit(lambda x: x + 1, 10)

TODO

  • [x] Deploy the scheduler on the cluster

  • [x] Support a command line interface

Docker Testing Harness

This sets up a docker cluster of one Mesos master and two Mesos agents using docker-compose.

Requires:

  • docker version >= 1.11.1

  • docker-compose version >= 1.7.1

docker-compose up

Run py.test:

py.test dask-marathon

Web UIs

History

Dask-marathon originally forked from https://github.com/mrocklin/dask-marathon

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

daskathon-1.3.1.tar.gz (23.3 kB view details)

Uploaded Source

File details

Details for the file daskathon-1.3.1.tar.gz.

File metadata

  • Download URL: daskathon-1.3.1.tar.gz
  • Upload date:
  • Size: 23.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for daskathon-1.3.1.tar.gz
Algorithm Hash digest
SHA256 1dfb096965e61337d0825ba578c787acd50022de5e96c31ee0a4cf4a82835f70
MD5 9f81d543d74e00aa7525dbbb04bd4c30
BLAKE2b-256 793ede77ae23ae88d4ab03c5f0575316987d3e758a824d914c2a12bc1805c22a

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page