marathon-acme 0.6.1

How it works

There is one big requirement for deploying marathon-acme: there must be shared persistent storage between marathon-acme and all marathon-lb instances. This will be used to store the certificates.

1. marathon-acme watches Marathon for changes to app definitions.

2. It collects the values of all MARATHON_ACME_{n}_DOMAIN labels on apps. This will form the set of domains to fetch certificates for.

3. It generates, verifies and stores certificates for any new domains using the configured ACME certificate authority.

4. It tells marathon-lb to reload using the marathon-lb HTTP API.

5. It issues new certificates for soon-to-expire certificates once a day.

marathon-acme is written in Python using Twisted. The certificate issuing functionality is possible thanks to the txacme library.

The ACME provider that most people are likely to use is Let’s Encrypt. Before using marathon-acme with Let’s Encrypt, make sure you are aware of their rate limits.

The entire certificate-issuing workflow is shown below:

Usage

marathon-acme is available as a pip-installable Python package on PyPI. However, most users will probably want to use the Docker image available from Docker Hub.

> $ docker run --rm praekeltfoundation/marathon-acme --help
usage: marathon-acme [-h] [-a ACME] [-e EMAIL] [-m MARATHON[,MARATHON,...]]
                     [-l LB[,LB,...]] [-g GROUP] [--allow-multiple-certs]
                     [--listen LISTEN] [--sse-timeout SSE_TIMEOUT]
                     [--log-level {debug,info,warn,error,critical}]
                     storage-dir

Automatically manage ACME certificates for Marathon apps

positional arguments:
  storage-dir           Path to directory for storing certificates

optional arguments:
  -h, --help            show this help message and exit
  -a ACME, --acme ACME  The address for the ACME Directory Resource (default:
                        https://acme-v01.api.letsencrypt.org/directory)
  -e EMAIL, --email EMAIL
                        An email address to register with the ACME service
                        (optional)
  -m MARATHON[,MARATHON,...], --marathon MARATHON[,MARATHON,...]
                        The addresses for the Marathon HTTP API (default:
                        http://marathon.mesos:8080)
  -l LB[,LB,...], --lb LB[,LB,...]
                        The addresses for the marathon-lb HTTP API (default:
                        http://marathon-lb.marathon.mesos:9090)
  -g GROUP, --group GROUP
                        The marathon-lb group to issue certificates for
                        (default: external)
  --allow-multiple-certs
                        Allow multiple certificates for a single app port.
                        This allows multiple domains for an app, but is not
                        recommended.
  --listen LISTEN       The address for the port to listen on (default: :8000)
  --sse-timeout SSE_TIMEOUT
                        Amount of time in seconds to wait for some event data
                        to be received from Marathon. Set to 0 to disable.
                        (default: 60)
  --log-level {debug,info,warn,error,critical}
                        The minimum severity level to log messages at
                        (default: info)
  --version             show program's version number and exit

{
  "id": "/marathon-acme",
  "cpus": 0.01,
  "mem": 128.0,
  "args": [
    "--email", "letsencrypt@example.com",
    "--marathon", "http://marathon1:8080,http://marathon2:8080,http://marathon3:8080",
    "--lb", "http://lb1:9090,http://lb2:9090",
    "/var/lib/marathon-acme"
  ],
  "labels": {
    "HAPROXY_GROUP": "external",
    "HAPROXY_0_VHOST": "marathon-acme.example.com",
    "HAPROXY_0_BACKEND_WEIGHT": "1",
    "HAPROXY_0_PATH": "/.well-known/acme-challenge/",
    "HAPROXY_0_HTTP_FRONTEND_ACL_WITH_PATH": "  acl host_{cleanedUpHostname} hdr(host) -i {hostname}\n  acl path_{backend} path_beg {path}\n  redirect prefix http://{hostname} code 302 if !host_{cleanedUpHostname} path_{backend}\n  use_backend {backend} if host_{cleanedUpHostname} path_{backend}\n"
  },
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "praekeltfoundation/marathon-acme",
      "network": "BRIDGE",
      "portMappings": [
        { "containerPort": 8000, "hostPort": 0 }
      ],
      "parameters": [
        {
          "value": "my-volume-driver",
          "key": "volume-driver"
        },
        {
          "value": "marathon-acme-certs:/var/lib/marathon-acme",
          "key": "volume"
        }
      ],
    }
  }
}

"labels": {
  "HAPROXY_GROUP": "external",
  "HAPROXY_0_VHOST": "marathon-acme.example.com",
  "HAPROXY_0_BACKEND_WEIGHT": "1",
  "HAPROXY_0_PATH": "/.well-known/acme-challenge/",
  "HAPROXY_0_HTTP_FRONTEND_ACL_WITH_PATH": "  acl host_{cleanedUpHostname} hdr(host) -i {hostname}\n  acl path_{backend} path_beg {path}\n  redirect prefix http://{hostname} code 302 if !host_{cleanedUpHostname} path_{backend}\n  use_backend {backend} if host_{cleanedUpHostname} path_{backend}\n"
}

external

marathon-acme.example.com

1

/.well-known/acme-challenge/

acl host_{cleanedUpHostname} hdr(host) -i {hostname}
acl path_{backend} path_beg {path}
redirect prefix http://{hostname} code 302 if !host_{cleanedUpHostname} path_{backend}
use_backend {backend} if host_{cleanedUpHostname} path_{backend}

"labels": {
  ...,
  "MARATHON_ACME_0_DOMAIN": "marathon-acme.example.com",
  "HAPROXY_0_HTTP_FRONTEND_ACL_WITH_PATH": "  acl host_{cleanedUpHostname} hdr(host) -i {hostname}\n  acl path_{backend} path_beg {path}\n  redirect prefix https://{hostname} code 302 if path_{backend}\n"
}

marathon-acme.example.com

acl host_{cleanedUpHostname} hdr(host) -i {hostname}
acl path_{backend} path_beg {path}
redirect prefix https://{hostname} code 302 if path_{backend}

--ssl-certs <storage-dir>/certs,<storage-dir>/default.pem

Limitations

The library used for ACME certificate management, txacme, is currently quite limited in its functionality. The two biggest limitations are:

• There is no Subject Alternative Name (SAN) support yet (#37). Each certificate will correspond to exactly one domain name. This limitation makes it easier to hit Let’s Encrypt’s rate limits.

• There is no support for removing certificates from txacme’s certificate store (#77). Once marathon-acme issues a certificate for an app it will try to renew that certificate forever unless it is manually deleted from the certificate store.

For a more complete list of issues, see the issues page for this repo.

Troubleshooting

Challenge ping endpoint

One common problem is that marathon-lb is misconfigured and ACME challenge requests are unable to reach marathon-acme. You can test challenge request routing to marathon-acme using the challenge ping endpoint.

It should be possible to reach the /.well-known/acme-challenge/ping path from all domains served by marathon-lb:

> $ curl cake-service.example.com/.well-known/acme-challenge/ping
{"message": "pong"}

> $ curl soda-service.example.com/.well-known/acme-challenge/ping
{"message": "pong"}