Skip to main content

A/B testing framework under active development at SeatGeek

Project description

https://travis-ci.org/seatgeek/sixpack.png?branch=master

Sixpack is a framework to enable A/B testing across multiple programming languages. It does this by exposing a simple API for client libraries. Client libraries can be written in virtually any language.

Sixpack has two main parts. The first, Sixpack-server, is responsible for responding to web requests. The second, Sixpack-web, is a web dashboard for tracking and acting on your A/B tests. Sixpack-web is optional.

Requirements

  • Redis

  • Python >= 2.7 (3.0 untested, pull requests welcome)

Getting Started

To get going, create (or don’t, but you really should) a new virtualenv for your sixpack installation. Follow that up with pip install:

$ pip install sixpack

Next, create a Sixpack configuration. A configuration must be created for sixpack to run. Here’s the default:

redis_port: 6379                        # Redis port
redis_host: localhost                   # Redis host
redis_prefix: sixpack                   # all Redis keys will be prefixed with this
redis_db: 15                            # DB number in redis

full_response: True                     # Not In Use
disable_whiplash: True                  # Disable the whiplash/multi-armed bandit choice Algorithm

# The regex to match for robots
robot_regex: $^|trivial|facebook|MetaURI|butterfly|google|amazon|goldfire|sleuth|xenu|msnbot|SiteUptime|Slurp|WordPress|ZIBB|ZyBorg|pingdom|bot|yahoo|slurp|java|fetch|spider|url|crawl|oneriot|abby|commentreader|twiceler
ignored_ip_addresses: []                # List of IP
control_on_db_failure: True             # Not in use
allow_multiple_experiments: False       # Not in Use

asset_path: gen                         # Path for compressed assets to live. This path is RELATIVE to sixpack/static
secret_key: '<your secret key here>'    # Random key (any string is valid, required for sixpack-web to run)

You can store this file anywhere (we recommend /etc/sixpack/config.yml). As long as Redis is running, you can now start the sixpack server like this:

$ SIXPACK_CONFIG=<path to config.yml> sixpack

Sixpack-server and Sixpack-web will be listening on ports 5000 and 5001, respectively. For use in a production environment, please see the “Production Notes” section below.

Using the API

All interaction with Sixpack is done via HTTP GET requests. Sixpack allows for cross-language testing by accepting a unique client_id (which the client is responsible for generating) that links a participation to a conversion. All requests to Sixpack require a client_id.

Participating in an Experiment

You can participate in an experiment with a GET request to the participate endpoint:

$ curl http://localhost:5000/participate?experiment=button_color&alternatives=red&alternatives=blue&alternatives=orange&client_id=12345678-1234-5678-1234-567812345678

If the test does not exist, it will be created automatically. You do not need to create the test in Sixpack-web.

Arguments

experiment (required) is the name of the test. Valid experiment names must be alphanumeric and can contain _ and -.

alternatives (required) are the potential responses from Sixpack. One of them will be the bucket that the client_id is assigned to.

client_id (required) is the unique id for the user participating in the test.

user_agent (optional) user agent of the user making a request. Used for bot detection.

ip_address (optional) ip address of user making a request. Used for bot detection.

force (optional) force a specific alternative to be returned, example:

$ curl http://localhost:5000/participate?experiment=button_color&alternatives=red&alternatives=blue&force=red&client_id=12345678-1234-5678-1234-567812345678

In this example, red will always be returned. This is used for testing only.

Response

A typical Sixpack participation response will look something like this:

{
    status: "ok",
    alternative: {
        name: "red"
    },
    experiment: {
        version: 0,
        name: "button_color"
    },
    client_id: "12345678-1234-5678-1234-567812345678"
}

The most interesting part of this is alternative. This is a representation of the alternative that was chosen for the test and assigned to a client_id. All subsequent requests to this experiment/client_id combination will be returned the same alternative.

Converting a user

You can convert a use with a GET request to the convert endpoint:

$ curl http://localhost:5000/convert?experiment=button_color&client_id=12345678-1234-5678-1234-567812345678

Arguments

experiment (required) the name of the experiment you would like to convert on

client_id (request) the client you would like to convert.

Notes

You’ll notice that the convert endpoint does not take a alternative query parameter. This is because Sixpack handles that internally with the client_id.

We’ve included a ‘health-check’ endpoint, available at /_status. This is helpful for monitoring and alerting if the Sixpack service becomes unavailable.

Clients

We’ve already provided clients in four languages. We’d love to add clients in additional languages. If you feel inclined to create one, please first read the CLIENTSPEC. After writing your client, please update and pull request this file so we know about it.

Dashboard

Sixpack comes with a built in dashboard. You can start the dashboard with:

$ SIXPACK_CONFIG=<path to config.yml> sixpack-web

The sixpack dashboard allows you to visualize how each experiment’s alternatives are doing compared to the rest, select alternatives as winners, and update experiment descriptions to something more human-readable

Production Notes

We recommend running Sixpack on gunicorn in production. You will need to install gunicorn in your virtual environment before running the following.

To run the sixpack server using gunicorn/gevent (a separate installation) you can run the following:

gunicorn --access-logfile - -w 8 --worker-class=gevent sixpack.server:start

To run the sixpack web dashboard using gunicorn/gevent (a separate installation) you can run the following:

gunicorn --access-logfile - -w 2 --worker-class=gevent sixpack.web:start

Contributing

  1. Fork it

  2. Start Sixpack in development mode with

$ PYTHONPATH=. SIXPACK_CONFIG=<path to config.yml> bin/sixpack

and

$ PYTHONPATH=. SIXPACK_CONFIG=<path to config.yml> bin/sixpack-web

We’ve also included a small script that will seed Sixpack with lots of random data for testing and development on sixpack-web. You can seed Sixpack with the following command

$ PYTHONPATH=. SIXPACK_CONFIG=<path to config.yml> sixpack/test/seed

This command will make a few dozen requests to the participate and convert endpoints. Feel free to run it multiple times to get additional data.

  1. Create your feature branch (git checkout -b my-new-feature)

  2. Write tests

  3. Commit your changes (git commit -am 'Added some feature')

  4. Push to the branch (git push origin my-new-feature)

  5. Create new pull request

Please avoid changing versions numbers; we’ll take care of that for you

Sixpack has a Google Group for support and discussion.

License

Sixpack is released under the BSD 2-Clause License.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

Sixpack-0.2.0.tar.gz (348.9 kB view details)

Uploaded Source

File details

Details for the file Sixpack-0.2.0.tar.gz.

File metadata

  • Download URL: Sixpack-0.2.0.tar.gz
  • Upload date:
  • Size: 348.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for Sixpack-0.2.0.tar.gz
Algorithm Hash digest
SHA256 78944cffbec456870324a12baa64ad6b137ae537609eb2d1ce810aa9c855863d
MD5 9e6fc94de3ac66540560a2dc5595c2f6
BLAKE2b-256 11d33e3c4fa102f404b9e2d6eb1fdcd7fdbe25e6c00122dd55c7249f092fb1cd

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page