fleece 0.19.0

Retry logging calls

A retry wrapper for logging handlers that occasionally fail is also provided. This wrapper can be useful in preventing crashes when logging calls to external services such as CloudWatch fail.

For example, consider the following handler for CloudWatch using watchtower:

logger.addHandler(
    watchtower.CloudWatchLogHandler(log_group='WORKER-POLL',
                                    stream_name=str(uuid.uuid4()),
                                    use_queues=False))

If the CloudWatch service is down, or rate limits the client, that will cause logging calls to raise an exception, which may interrupt the script. To avoid that, the watchtower handler can be wrapped in a RetryHandler as follows:

logger.addHandler(
    fleece.log.RetryHandler(
        watchtower.CloudWatchLogHandler(log_group='WORKER-POLL',
                                        stream_name=str(uuid.uuid4()),
                                        use_queues=False)))

In the above example, logging calls that fail will be retried up to 5 times, using an exponential backoff algorithm to increasingly space out retries. If all retries fail, then the logging call will, by default, give up silently and return, allowing the program to continue. See the documentation for the RetryHandler class for information on how to customize the retry strategy.

Logging request IDs for each log event

It might be very helpful to have the API Gateway and/or Lambda request IDs present in each log event, so that troubleshooting problematic requests becomes easy. If you apply the fleece.log.inject_request_ids_into_environment decorator to the Lambda handler function, an api_request_id (only if the event source is API Gateway) and a lambda_request_id attribute will be added to the log event dictionary.

Prerequisites

1. Make sure you add the following permissions to the Lambda execution role of your function: xray:PutTraceSegments and xray:PutTelemetryRecords.

2. Enable active tracing under Advanced settings on the Configuration tab of your Lambda function in the AWS Console (or using the ``update_function_configuration` API call <http://boto3.readthedocs.io/en/latest/reference/services/lambda.html#Lambda.Client.update_function_configuration>`_).

Features

You can mark any function or method for tracing by using the @trace_xray_subsegment decorator. You can apply the decorator to any number of functions and methods, the resulting trace will be properly nested. You have to decorate all the methods you want traced (e.g. if you decorate your handler function only, no other functions will be traced that it calls).

This module also provides wrappers for boto and requests so that any AWS API call, or HTTP request will be automatically traced by X-Ray, but you have to explicitly allow this behavior by calling monkey_patch_botocore_for_xray and/or monkey_patch_requests_for_xray. The best place to do this would be the main handler module where the Lambda entry point is defined.

A quick example (handler.py)

from fleece import boto3
from fleece.xray import (monkey_patch_botocore_for_xray,
                         trace_xray_subsegment)

monkey_patch_botocore_for_xray()


@trace_xray_subsegment()
def lambda_handler(event, context):
    return get_user()


def get_user():
    # This function doesn't have to be decorated, because the API call to IAM
    # will be traced thanks to the monkey-patching.
    iam = boto3.client('iam')
    return iam.get_user()

Note: the monkey-patched tracing will also work with the wrappers described above.

Installation

pip install fleece[cli]

fleece build

usage: fleece build [-h] [--python36] [--rebuild]
                    [--requirements REQUIREMENTS]
                    [--dependencies DEPENDENCIES] [--target TARGET]
                    [--source SOURCE]
                    [--exclude [EXCLUDE [EXCLUDE ...]]]
                    service_dir

Simple Lambda builder.

positional arguments:
  service_dir           directory where the service is located (default: $pwd)

optional arguments:
  -h, --help            show this help message and exit
  --python36, -3        use Python 3.6 (default: Python 2.7)
  --rebuild             rebuild Python dependencies
  --requirements REQUIREMENTS, -r REQUIREMENTS
                        requirements.txt file with dependencies (default:
                        $service_dir/src/requirements.txt)
  --dependencies DEPENDENCIES, -d DEPENDENCIES
                        comma separated list of system dependencies
  --target TARGET, -t TARGET
                        target directory for lambda_function.zip (default
                        $service_dir/dist)
  --source SOURCE, -s SOURCE
                        source directory to include in lambda_function.zip
                        (default: $service_dir/src)
  --exclude [EXCLUDE [EXCLUDE ...]], -e [EXCLUDE [EXCLUDE ...]]
                        glob pattern to exclude

To build a lambda package from the service’s top-level directory:

$ fleece build .

The assumptions made with the above command are that the source code of the service is in ./src, the requirements file is in ./src/requirements.txt and the output zip file will be written to ./dist. These defaults can be changed with the --source, --requirements and --target options respectively.

The build process will run in a Docker container based on the Amazon Linux image. If there are any additional dependencies that need to be installed on the container prior to installing the Python requirements, those can be given with the --dependencies option. Any environment variables recognized by pip, such as PIP_INDEX_URL, are passed on to the container.

fleece run

usage: fleece run [-h] [--username USERNAME] [--apikey APIKEY]
                  [--config CONFIG] [--account ACCOUNT]
                  [--environment ENVIRONMENT] [--role ROLE]
                  command

Run command in environment with AWS credentials from Rackspace FAWS API

positional arguments:
  command               Command to execute

optional arguments:
  -h, --help            show this help message and exit
  --username USERNAME, -u USERNAME
                        Rackspace username. Can also be set via RS_USERNAME
                        environment variable
  --apikey APIKEY, -k APIKEY
                        Rackspace API key. Can also be set via RS_API_KEY
                        environment variable
  --config CONFIG, -c CONFIG
                        Path to YAML config file with defined accounts and
                        aliases. Default is ./environments.yml
  --account ACCOUNT, -a ACCOUNT
                        AWS account number. Cannot be used with
                        `--environment`
  --environment ENVIRONMENT, -e ENVIRONMENT
                        Environment alias to AWS account defined in config
                        file. Cannot be used with `--account`
  --role ROLE, -r ROLE  Role name to assume after obtaining credentials from
                        FAWS API

# fleece run --username $username --apikey $apikey --account $account 'aws s3 ls'
2017-10-02 12:03:18 bucket1
2017-06-08 14:31:07 bucket2
2017-08-10 17:28:47 bucket3
2017-08-10 17:21:58 bucket4
2017-08-15 20:33:02 bucket5

You can also setup an environments file to reduce command-line flags:

# cat environments.yml
environments:
  - name: development
    account: '123456789012'
  - name: staging
    account: '123456789012'
    rs_username_var: MY_RS_USERNAME
    rs_apikey_var: MY_RS_APIKEY
  - name: testing
    account: '123456789012'
  - name: production
    account: '123456789012'
    role: LambdaDeployRole

# fleece run --username $username --apikey $apikey --environment testing 'aws s3 ls'
2017-10-02 12:03:18 bucket1
2017-06-08 14:31:07 bucket2
2017-08-10 17:28:47 bucket3
2017-08-10 17:21:58 bucket4
2017-08-15 20:33:02 bucket5

Note the staging environment example above, which provides a custom pair of environment variables from where the Rackspace username and API key are sourced. These would be used only if credentials are not explicitly given as part of the command.

fleece config

usage: fleece config [-h] [--config CONFIG] [--username USERNAME]
                     [--apikey APIKEY] [--environments ENVIRONMENTS]
                     {import,export,edit,render} ...

Configuration management

positional arguments:
  {import,export,edit,render}
                        Sub-command help
    import              Import configuration from stdin
    export              Export configuration to stdout
    edit                Edit configuration
    render              Render configuration for an environment

optional arguments:
  -h, --help            show this help message and exit
  --config CONFIG, -c CONFIG
                        Config file (default is config.yml)
  --username USERNAME, -u USERNAME
                        Rackspace username. Can also be set via RS_USERNAME
                        environment variable
  --apikey APIKEY, -k APIKEY
                        Rackspace API key. Can also be set via RS_API_KEY
                        environment variable
  --environments ENVIRONMENTS, -e ENVIRONMENTS
                        Path to YAML config file with defined accounts and
                        environment names. Defaults to ./environments.yml

The fleece config command has a few sub-commands that work with configuration files. There are a number of arguments that apply to all commands:

• --config sets the configuration file. This is the file that holds the configuration, in a format that is appropriate to commit to source control (i.e. sensitive variables are encrypted).

• --username and --apikey are the Rackspace credentials, used to obtain temporary AWS access credentials from FAWS. For convenience, these can be set in environment variables.

• --environments is an environments.yml file that defines the different environments and the associated AWS accounts for each. The format is as described in the fleece run command.

The config commands work with two types of config files. The config.yml file is a “closed” config file, where all sensitive values are encrypted. Developers typically do not edit this file but instead export it to a temporary “open” configuration file where sensitive variables appear in plain text for editing. As soon as changes are made, the open config file is imported back into the closed config.yml.

The open configuration format is as follows:

stages:                                 # stage definitions
  prod:                                 # stage name
    environment: prod                   # environment associated with this stage
    key: prod-key-here                  # KMS key, ARN or name with or without the "alias/" prefix are all valid
  /.*/:                                 # regular expressions for custom stage names
    environment: dev
    key: dev-key-here
config:
  foo: bar                              # plain text variable
  password:                             # per-stage values, encrypted
    +dev: :encrypt:my-dev-password      # per-stage keys must have a "+" prefix so they are
    +prod: :encrypt:my-prod-password    # not taken as a nested dict
    +/.*/: :encrypt:my-custom-password
  nested:                               # nested dictionaries
    inner_var: value
    a_list:                             # list of dictionaries
      - username1:                      # per-stage values, without encryption
          +prod: bob-prod
          +/.*/: bob-dev
        password1:                      # per-stage values, encrypted
          +prod: :encrypt:bob-prod-pw
          +/.*/: :encrypt:bob-dev-pw
      - username2: user2
        password2:
          +prod: :encrypt:prod-pw2
          +/.*/: :encrypt:dev-pw2

The stages section defines the available stages, along with their association to an environment and a KMS key. The environment, which must be defined in the environments.yml, links the stage to a AWS account. The KMS key can be given as an ARN or as an alias. The alias can be given with or without the alias/ prefix. Stage names can be given explicitly or as a regular expression (surrounded by /s). When fleece needs to match a stage name given in one of its commands, it will first attempt to do an equality match, and only when that fails it will try the regular expression based stage names. The regular expression stage names are evaluated in random order until one succeeds, so it is important to avoid ambiguities in the regex patterns.

The config section is where configuration variables are defined. A standard key/value pair in this section represents a plaintext variable that will be made available for all stages. A variable can be given per-stage values by making its value a sub-dictionary where the keys are the stage names prefixed by +. Regex patterns for stage names are supported here as well.

Any variables that are sensitive and need to be encrypted must have per-stage values, and these values must have the :encrypt: prefix so that fleece knows to encrypt them when the configuration is imported and stored in config.yml.

The available sub-commands are:

ENCRYPTED_CONFIG = ['... encrypted blob here ...']
import base64
import boto3
import json

def load_config():
    config_json = ''
    kms = boto3.client('kms')
    for buffer in ENCRYPTED_CONFIG:
        r = kms.decrypt(CiphertextBlob=base64.b64decode(buffer.encode(
            'utf-8')))
        config_json += r['Plaintext'].decode('utf-8')
    return json.loads(config_json)

CONFIG = load_config()

from fleece_config import CONFIG