coal-mine 0.4.17

Operations

The operations that can be performed on canaries through the CLI or API are:

• create

• delete

• reconfigure

• get information about

• pause – stop monitoring and alerting

• unpause

• trigger

• list – all canaries or the ones matching search terms

Coal Mine security is rudimentary. If the server is configured with an optional authentication key, then the key must be specified with all operations except trigger.

Data

These canary attributes are specified when it is created or updated:

• name

• description

• periodicity – the maximum number of seconds that can elapse before a canary is late, or a schedule in the format described below, which allows the periodicity of the canary to vary over time

• zero or more notification email address(es)

These are created and maintained by Coal Mine:

• slug – the canary’s name, lower-cased, with spaces and underscores converted to hyphens and other non-alphanumeric characters removed

• a random identifier consisting of eight lower-case letters, generated when the canary is created and guaranteed to be unique against other canaries in the database

• late state (boolean)

• paused state (boolean)

• deadline by which the canary should be triggered to avoid being late

• a history of triggers, pruned when >1000 or (>100 and older than one week)

# 5-minute delays are ok on weekends ;
* * * * sat,sun 300 ;
# 5-minute days are ok overnight ;
* 0-12 * * mon-fri 300 ;
# otherwise, we require a shorter periodicity ;
* 13-23 * * mon-fri 90

Server

1. pip install coal-mine

2. Create /etc/coal-mine.ini (see below)

3. Run coal-mine &

4. Put that in /etc/rc.local or something as needed to ensure that it is restarted on reboot.

CLI

1. pip install coal-mine

2. cmcli configure [--host server-host-name] [--port server-port] [--auth-key key | --no-auth-key]

The --host argument can take a URL base (i.e., http://server-host-name or https://server-host-name) as well. This is useful if, for example, you’ve put your Coal Mine server behind an SSL proxy so the CLI needs to use SSL to connect to it.

The CLI stores its configuration in ~/.coal-mine.ini. Note that the authentication key is stored in plaintext. Any configuration parameters the CLI needs that aren’t stored in the INI file must be specified explicitly on the command line when using the CLI.

Example commands

$ coal-mine &
[1] 7564
$ curl 'http://coal-mine-server/coal-mine/v1/canary/create?name=My+First+Canary&periodicity=3600'
{
    "status": "ok",
    "canary": {
        "deadline": "2015-03-19T02:08:44.885182",
        "id": "fbkvlsby",
        "paused": false,
        "description": "",
        "periodicity": 3600,
        "name": "My First Canary",
        "slug": "my-first-canary",
        "emails": [],
        "history": [
            [
                "2015-03-19T01:08:44.885182",
                "Canary created"
            ]
        ],
        "late": false
    }
}
$ curl 'http://coal-mine-server/fbkvlsby?comment=short+form+trigger+url'
{
    "recovered": false,
    "unpaused": false,
    "status": "ok"
}
$ curl 'http://coal-mine-server/coal-mine/v1/canary/trigger?slug=my-first-canary&comment=long+form+trigger+url'
{
    "recovered": false,
    "unpaused": false,
    "status": "ok"
}
$ curl 'http://coal-mine-server/coal-mine/v1/canary/get?name=My+First+Canary'
{
    "canary": {
        "paused": false,
        "name": "My First Canary",
        "history": [
            [
                "2015-03-19T01:11:56.408000",
                "Triggered (long form trigger url)"
            ],
            [
                "2015-03-19T01:10:42.608000",
                "Triggered (short form trigger url)"
            ],
            [
                "2015-03-19T01:08:44.885000",
                "Canary created"
            ]
        ],
        "emails": [],
        "id": "fbkvlsby",
        "late": false,
        "slug": "my-first-canary",
        "deadline": "2015-03-19T02:11:56.408000",
        "periodicity": 3600,
        "description": ""
    },
    "status": "ok"
}

All API endpoints are fully documented below.

Watching a cron job

0 0 * * * my-backup-script.sh && (curl http://coal-mine-server/fbkvlsby &>/dev/null)

Create canary

Endpoint: /coal-mine/v1/canary/create

Side effects:

Adds canary to database. Creates history record at current time with “Canary created” as its comment. Sets deadline to current time plus periodicity, unless “paused” was specified.

Required parameters:

• name

• periodicity

• auth_key (if authentication is enabled in the server)

Optional parameters:

• description - empty if unspecified

• email - specify multiple times for multiple addresses; no notifications if unspecified

• paused - allows canary to be created already in paused state

Response is the same as shown for get().

Delete canary

Endpoint: /coal-mine/v1/canary/delete

Required parameters:

• name, id, or slug

• auth_key

Response:

{'status': 'ok'}

Update canary

Endpoint: /coal-mine/v1/canary/update

Side effects:

Updates the specified canary attributes. Updates deadline to latest history timestamp plus periodicity if periodicity is updated and canary is unpaused, and sets late state if new deadline is before now. Sends notification if canary goes from not late to late or vice versa.

Required parameters:

• id or slug (not name, which should only be specified to update the name and slug)

• auth_key

Optional parameters:

• name

• periodicity

• description

• email - specify a single value of “-” to clear existing email addresses

Response is the same as shown for get().

Get canary

Endpoint: /coal-mine/v1/canary/get

Required parameters:

• name, id, or slug

• auth_key

Response:

{'status': 'ok',
 'canary': {'name': name,
           'description': description,
           'id': identifier,
           'slug': slug,
           'periodicity': seconds,
           'emails': [address, ...],
           'late': boolean,
           'paused': boolean,
           'deadline': 'YYYY-MM-DDTHH:MM:SSZ',
           'history': [['YYYY-MM-DDTHH:MM:SSZ', comment], ...]}}

List canaries

Endpoint: /coal-mine/v1/canary/list

Required parameters:

• auth_key

Optional parameters:

• verbose - include all query output for each canary

• paused - boolean, whether to list paused / unpaused canaries only

• late - boolean, whether to list late / timely canaries only

• search - string, regular expression to match against name, identifier, and slug

Response:

{'status': 'ok',
 'canaries': [{'name': name,
             'id': identifier},
            ...]}

If “verbose” is true, then the JSON for each canary includes all the fields shown above, not just the name and identifier.

Trigger canary

Endpoint: /coal-mine/v1/canary/trigger

Also: /identifier, in which case the “id” parameter is implied

Note that the server will accept POST requests for triggers as well as GET requests, so that you can use triggers as webhooks in applications that expect to be able to POST. The content of the POST is ignored; even when using POST, the API parameters must still be specified as a query string.

Side effects:

Sets late state to false. Sets deadline to now plus periodicity. Adds history record. Prunes history records. Unpauses canary. Generates notification email if canary was previously late.

Required parameters:

• name, id, or slug

Optional parameters:

• comment - stored in history with trigger record

Response:

{'status': 'ok', 'recovered': boolean, 'unpaused': boolean}

• recovered - indicates whether the canary was previously late before this trigger

• unpaused - indicates whether the canary was previously paused before this trigger

Pause canary

Endpoint: /coal-mine/v1/canary/pause

Side effects:

Clears deadline. Sets late state to false if necessary. Pauses canary. Adds history record about pause. Prunes history records.

Required parameters:

• name, id, or slug

• auth_key

Optional parameters:

• comment

Response is the same as shown for get().

Unpause canary

Endpoint: /coal-mine/v1/canary/unpause

Side effects:

Sets deadline to now plus periodicity. Unpauses canary. Adds history record about unpause. Prunes history records.

Required parameters:

• name, id, or slug

• auth_key

Optional parameters:

• comment

Response is the same as shown for get().

To Do

(Pull requests welcome!)

Other storage engines.

Other notification mechanisms.

More smtplib configuration options in INI file.

Web UI.

Links to Web UI in email notifications.

Repeat notifications if a canary remains late for an extended period of time? Not even sure I want this.

Better authentication?

Support time-zone localization of displayed timestamps.

SSL support in server