daskathon 1.1

Run

It’s not yet clear how to expose all of the necessary options to a command line interface. For now we’re doing everything manually.

Make an IOLoop running in a separate thread:

with MarathonCluster(marathon='http://localhost:8080',
                     cpus=1, mem=512, adaptive=True) as mc:
    with Client(mc.scheduler_address) as c:
        x = c.submit(lambda x: x + 1, 1)
        assert x.result() == 2

Create a Client and submit work to the scheduler. Marathon will scale workers up and down as neccessary in response to current workload.

from distributed import Client
c = Client(s.address)

future = c.submit(lambda x: x + 1, 10)

TODO

• [x] Deploy the scheduler on the cluster

• [x] Support a command line interface

Docker Testing Harness

This sets up a docker cluster of one Mesos master and two Mesos agents using docker-compose.

Requires:

• docker version >= 1.11.1

• docker-compose version >= 1.7.1

docker-compose up

Run py.test:

py.test dask-marathon