sofine 0.2.2

What Problem Does sofine solve?

You need to get data related to a set of keys from many sources: web scrapers, Web APIs, flat files, data stores. Wouldn’t it be nice to build one combined data set over multiple calls with one command line, REST or Python call? Wouldn’t it be great if each data retrieval script you wrote was a reusable plugin that you could combine with any other?

You need a “glue API.”

This is the problem sofine solves. It’s a small enough problem that you could solve it yourself. But sofine is minimal to deploy and write plugins for, and has already decided in an optimally flexible way the same design decisions you would have to make if you wrote this yourself.

Features

1. Do (almost) no more work than if you wrote one-off data collection scripts

2. Manage your data retrieval plugins in any directory with any directory structure you like

3. Call plugins from the command line, as REST resources or from Python

4. Chain as many plugin calls as you want together and get back one JSON data set with all the data collected from all the chained calls

5. If called from the command line, sofine reads data from stdin if it is present, and always outputs to stdout. So sofine piped calls can themselves be composed in larger piped expressions.

For fun, here is an example of features 4 and 5, combining a sofine pipeline with the fantastic JSON query tool jq for further filtering.

echo '{"AAPL":[]}' | python $PYTHONPATH/sofine/runner.py '--SF-s ystockquotelib --SF-g example | --SF-s google_search_results --SF-g example' | jq 'map(recurse(.results) | {titleNoFormatting}'

Overview

To get started, you:

1. pip install sofine

2. Make sure your $PYTHONPATH points to the package directory where pip installed sofine

3. Create a plugin directory and assign it’s path to the environment SOFINE_PLUGIN_PATH

4. Write and call some data retrieval plugins (or just start using the included ones)

Plugins require two attributes and one method in the simple case and three methods in the most elaborate edge case. You can optionally define two additional attributes for clients to use to introspect your plugins.

sofine ships with a few useful plugins to get you started and give you the idea; you can combine these with your custom plugins with no additional configuration or code. The included plugins are:

• sofine.plugins.standard.file_source - Retrieves keys from a JSON file to add to the data set being built. See here for the details.

• example.archive_dot_org_search_results - Takes a search query and returns results from www.archive.org

• example.google_search_results - Takes a search query and returns results from the Google Search API

• example.fidelity - Takes a userId, pin, accountId and email, logs into Fidelity, scrapes the account portfolio and returns the tickers found as keys and four attributes of data for each ticker

• example.ystockquotelib - Takes a list of tickers and returns the data available from Yahoo! Finance for each ticker

Here is what usage looks like …

From the command line:

$ echo '{"AAPL":[]}' | python $PYTHONPATH/sofine/runner.py '--SF-s ystockquotelib --SF-g example | --SF-s google_search_results --SF-g example'

REST-fully:

$ python $PYTHONPATH/sofine/rest_runner.py

$ curl -X POST -d '{"AAPL":[]}' --header "Content-Type:application/json" http://localhost:10000/SF-s/ystockquotelib/SF-g/example/SF-s/google_search_results/SF-g/example

From Python:

import sofine.runner as runner

data = {"AAPL": []}
data_sources = ['ystockquotelib', 'google_search_results']
data_source_groups = ['example', 'example']
data_source_args = [[], []]
data = runner.get_data_batch(data, data_sources, data_source_groups, data_source_args)

All three calling styles return the same data set. sofine data sets map string keys to arrays of attributes, which are Python dicts. By default, these are returned as JSON to stdout. sofine also ships with support for CSV, and you can write your own data format plugins (more on that below).

Here is an example retrieved using included data retrieval plugins: the key “AAPL,” with all the attributes retrieved from Yahoo! Finance and the Google Search API combined.

{
"AAPL":
    [
        {
            "results": [
            {
                "GsearchResultClass": "GwebSearch",
                "cacheUrl": "http://www.google.com/search?q=cache:XhbIlCyrcXMJ:finance.yahoo.com",
                "content": "View the basic <b>AAPL</b> stock chart on Yahoo! Finance. Change the date range, chart type and compare Apple Inc. against other companies.",
                "title": "<b>AAPL</b>: Summary for Apple Inc.- Yahoo! Finance",
                "titleNoFormatting": "AAPL: Summary for Apple Inc.- Yahoo! Finance",
                "unescapedUrl": "http://finance.yahoo.com/q?s=AAPL",
                "url": "http://finance.yahoo.com/q%3Fs%3DAAPL",
                "visibleUrl": "finance.yahoo.com"
            },
            ...
            ...
            ]
        },
        {"avg_daily_volume": "59390100"},
        {"book_value": "20.193"},
        {"change": "+1.349"},
        {"dividend_per_share": "1.7771"},
        {"dividend_yield": "1.82"},
        {"earnings_per_share": "6.20"},
        {"ebitda": "59.128B"},
        {"fifty_day_moving_avg": "93.8151"},
        {"fifty_two_week_high": "99.24"},
        {"fifty_two_week_low": "63.8886"},
        {"market_cap": "592.9B"},
        {"price": "99.02"},
        {"price_book_ratio": "4.84"},
        {"price_earnings_growth_ratio": "1.26"},
        {"price_earnings_ratio": "15.75"},
        {"price_sales_ratio": "3.28"},
        {"short_ratio": "1.70"},
        {"stock_exchange": "\"NasdaqNM\""},
        {"two_hundred_day_moving_avg": "82.8458"},
        {"volume": "55317688"}
    ]
}

Installing sofine

pip install sofine

Then, make sure your $PYTHONPATH variable is set and points to the site-packages directory of your Python where pip installed sofine.

export PYTHONPATH=<MY PYTHON SITE-PACKAGES DIRECTORY>

Then, create a plugin directory and assign its path to an environment variable SOFINE_PLUGIN_PATH. You probably want to add it to your shell configuration file.

export SOFINE_PLUGIN_PATH=<MY PATH>

sofine runs its REST server on port 10000. If you want to use a different port, set the environment variable SOFINE_REST_PORT. You probably want to add it to your shell configuration file.

export SOFINE_REST_PORT=<MY PORT>

If you are going to create data format plugins, create a data format lugin directory and assign its path to the environment variable SOFINE_DATA_FORMAT_PLUGIN_PATH.

export SOFINE_DATA_FORMAT_PLUGIN_PATH=<MY PATH>

If you want to use the included fidelity and ystockquotelib plugins in the plugins.examples plugin group, also install the following:

easy_install mechanize
easy_install beautifulsoup4
pip install ystockquote

Two Kinds of Plugins: Data Retrieval and Data Format

sofine uses two kinds of plugins. Data retrieval plugins are what you call singly or in chained expressions to return data sets. When the documentation says “plugin,” it means data retrieval plugin. But sofine also supports plugins for the data format of data sets. By default sofine expects input on stdin in JSON format and writes JSON to stdout. But there is also a plugin for CSV.

How Data Retrival Plugins Work and How to Write Them

class ArchiveDotOrgSearchResults(plugin_base.PluginBase):
    def __init__(self):
        super(ArchiveDotOrgSearchResults, self).__init__()

plugin = ArchiveDotOrgResults

self.schema = ['quote']

def get_data(self, keys, args):
    """
    * `keys` - `list`. The list of keys to process.
    * `args` - `'list`. Empty for this plugin.
    Calls the Yahoo API to get all available fields for each ticker provided as a key in `keys`."""
    return {ticker : ystockquote.get_all(ticker) for ticker in keys}

{"trading::get_quotes::quote" : 47.65}

def parse_args(self, argv):
    """`[-p|--path]` - Path to the file listing the keys to load into this data source."""

    usage = "[-p|--path] - Path to the file listing the keys to load into this data source."
    parser = OptionParser(usage=usage)
    parser.add_option("-p", "--path",
                    action="store", dest="path",
                    help="Path to the file listing the keys to load into this data source. Required.")
    (opts, args) = parser.parse_args(argv)

    is_valid = True
    if not opts.path:
        print "Invalid argument error."
        print "Your args: path {0}".format(opts.path)
        print usage
        is_valid = False

    return is_valid, [opts.path]

import urllib
import urllib2
import json

def query_google_search(k):
    url = 'http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q={0}'.format(urllib.quote(k))
    ret = urllib2.urlopen(url)
    ret = ret.read()
    ret = json.loads(ret)

    if ret: 2
        ret = {'results' : ret['responseData']['results']}
    else:
        ret = {'results' : []}

    return ret

from sofine.plugins import plugin_base as plugin_base

class GoogleSearchResults(plugin_base.PluginBase):

    def __init__(self):
        super(GoogleSearchResults, self).__init__()
        self.name = 'google_search_results'
        self.group = 'example'
        self.schema = ['results']
        self.adds_keys = False

    def get_data(self, keys, args):
        return {k : query_google_search(k) for k in keys}

plugin = GoogleSearchResults

from sofine.plugins import plugin_base as plugin_base
import ystockquote

class YStockQuoteLib(plugin_base.PluginBase):

    def __init__(self):
        super(YStockQuoteLib, self).__init__()
        self.name = 'ystockquotelib'
        self.group = 'example'
        self.schema = ['fifty_two_week_low', 'market_cap', 'price', 'short_ratio',
                       'volume','dividend_yield', 'avg_daily_volume', 'ebitda',
                       'change', 'dividend_per_share', 'stock_exchange',
                       'two_hundred_day_moving_avg', 'fifty_two_week_high',
                       'price_sales_ratio', 'price_earnings_growth_ratio',
                       'fifty_day_moving_avg', 'price_book_ratio', 'earnings_per_share',
                       'price_earnings_ratio', 'book_value']
        self.adds_keys = False

    def get_data(self, keys, args):
        return {ticker : ystockquote.get_all(ticker) for ticker in keys}

plugin = YStockQuoteLib

How Data Format Plugins Work and How to Write Them

sofine defaults to expecting input and returning output in JSON format. The library also includes a CSV data format plugin. If these don’t meet your needs you can write your own, deploy them in your SOFINE_DATA_FORMAT_PLUGIN_PATH plugin directory and the use them by passing an additional data format argument in your calls.

• deserialize(data) - converts data in the data format to a Python data structure

• serialize(data) - converts a Python data structure to the data format

• get_content_type() - returns the correct value for the HTTP Content-Type header for the data format

The included format_json plugin provides a trivial example:

import json

def deserialize(data):
    return json.loads(data)

def serialize(data):
    return json.dumps(data)

def get_content_type():
    return 'application/json'

Formats without an isomorphic mapping to Python dicts and lists (which correspond to JSON objects and arrays) require some implementation. Specifically, your plugin needs to be aware of the sofine data structure for its data retrieval data sets, so that it can convert from the data format into that Python data structure in deserialize anc convert from that Python data structure into your data format (in a way that makes sense and is documented in your plugin) in serialize.

Remember, sofine data sets look like this:

{ “AAPL”: [ { “results”: [ { “GsearchResultClass”: “GwebSearch”, …

            },
            ...
            ]
        },
        {"avg_daily_volume": "59390100"},
        {"book_value": "20.193"},
        ...
    ]
}

As an example, here are the two methods in the included format_csv plugin:

def deserialize(data):
    ret = {}
    schema = []

    reader = csv.reader(data.split(lineterminator), delimiter=delimiter, i
                        lineterminator='', quoting=quoting, quotechar=quotechar)

    for row in reader:
        if not len(row):
            continue

        # 0th elem in CSV row is data row key
        key = row[0]
        key.encode('utf-8')

        attr_row = row[1:]
        ret[key] = [{attr_row[j].encode('utf-8') : attr_row[j + 1].encode('utf-8')}
                    for j in range(0, len(attr_row) - 1, 2)]

    return ret


def serialize(data):
    out_strm = BytesIO()
    writer = csv.writer(out_strm, delimiter=delimiter, lineterminator='|',
                        quoting=quoting, quotechar=quotechar)

    # Flatten each key -> [attrs] 'row' in data into a CSV row with
    #  key in the 0th position, and the attr values in an array in fields 1 .. N
    for key, attrs in data.iteritems():
        row = []
        row.append(key)
        for attr in attrs:
            row.append(attr.keys()[0])
            row.append(attr.values()[0])
        writer.writerow(row)

    ret = out_strm.getvalue()
    out_strm.close()

    return ret

How to Call Data Retrieval Plugins

As we saw above in the Introduction section, there are three ways to call plugins, from the command line, as REST resources, or in Python. When calling plugins to retrieve data, you need to pass three or four arguments, data, the plugin name, the plugin group and the plugin action.

There are six actions, which correspond to the five methods get_data, get_namespaced_data, parse_args, get_schema and get_namespaced_schema, while adds_keys returns the value of the the plugin’s self.adds_keys.

get_data
get_namespaced_data
parse_args
get_schema
get_namespaced_schema
adds_keys

python $PYTHONPATH/sofine/runner.py '--SF-s fidelity --SF-g example -c <CUSTOMER_ID> -p <PIN> -a <ACCOUNT_ID> -e <EMAIL> | --SF-s ystockquotelib --SF-g example'

curl -X POST -d '{}' --header "Content-Type:application/json" http://localhost:10000/SF-s/fidelity/SF-g/example/c/<CUSTOMER_ID>/p/<PIN>/a/<ACCOUNT_ID>/e/<EMAIL>/SF-s/ystockquotelib/SF-g/example

import sofine.runner as runner

data = {}
data_sources = ['fidelity', 'ystockquotelib']
data_source_groups = ['example', 'example']
data_source_args = [[customer_id, pin, account_id, email], []]
data = runner.get_data_batch(data, data_sources, data_source_groups, data_source_args)

{
    "key_1": [{"attribute_1": value_1}, {"attribute_2": value_2}, ...],
    "key_2": ...
}

python $PYTHONPATH/sofine/runner.py '--SF-d format_csv --SF-s fidelity --SF-g example -c <CUSTOMER_ID> -p <PIN> -a <ACCOUNT_ID> -e <EMAIL> | --SF-s ystockquotelib --SF-g example'

python $PYTHONPATH/sofine/runner.py '--SF-s fidelity --SF-g example --SF-a get_namespaced_data -c <CUSTOMER_ID> -p <PIN> -a <ACCOUNT_ID> -e <EMAIL> | --SF-s ystockquotelib --SF-g example --SF-a get_namespaced_data'

curl -X POST -d '{}' --header "Content-Type:application/json" http://localhost:10000/SF-s/fidelity/SF-g/example/SF-a/get_namespaced_data/c/<CUSTOMER_ID>/p/<PIN>/a/<ACCOUNT_ID>/e/<EMAIL>/SF-s/ystockquotelib/SF-g/example/SF-a/get_namespaced_data

{
    "key_1": [{"plugin_group::plugin_name::attribute_1": value_1},
    i         {"plugin_group::plugin_name::attribute_2": value_2}, ...],
    "key_2": ...
}

import sofine.runner as runner

data = {}
data_sources = ['fidelity', 'ystockquotelib']
data_source_groups = ['example', 'example']
data_source_args = [[customer_id, pin, account_id, email], []]
data = runner.get_data_batch(data, data_sources, data_source_groups, data_source_args)

python $PYTHONPATH/sofine/runner.py '--SF-s file_source --SF-g standard --SF-a parse_args -p "./sofine/tests/fixtures/file_source_test_data.txt"'

curl -X POST -d '{}' --header "Content-Type:application/json" http://localhost:10000/SF-s/file_source/SF-g/standard/SF-a/parse_args/p/.%2Fsofine%2Ftests%2Ffixtures%2Ffile_source_test_data.txt

def test_parse_args_file_source(self):
    data_source = 'file_source'
    data_source_group = 'standard'
    path = './sofine/tests/fixtures/file_source_test_data.txt'
    args = ['-p', path]
    actual = runner.parse_args(data_source, data_source_group, args)

    self.assertTrue(actual['is_valid'] and actual['parsed_args'] == [path])

{"is_valid": true|false, "parsed_args": [arg_1, arg_2, ...]}

python $PYTHONPATH/sofine/runner.py '--SF-s ystockquotelib --SF-g example --SF-a get_schema'

curl -X POST -d '{}' --header "Content-Type:application/json" http://localhost:10000/SF-s/ystockquotelib/SF-g/example/SF-a/get_schema

data_source = 'ystockquotelib'
data_source_group = 'example'
schema = runner.get_schema(data_source, data_source_group)

{"schema": [attribute_key_name_1, attribute_key_name_2, ...]}

python $PYTHONPATH/sofine/runner.py '--SF-s ystockquotelib --SF-g example --SF-a get_namespaced_schema'

curl -X POST -d '{}' --header "Content-Type:application/json" http://localhost:10000/SF-s/ystockquotelib/SF-g/example/SF-a/get_namespaced_schema

data_source = 'ystockquotelib'
data_source_group = 'example'
schema = runner.get_namespaced_schema(data_source, data_source_group)

{
    "schema": [plugin_group::plugin_name::attribute_key_name_1,
               plugin_group::plugin_name::attribute_key_name_2, ...]
}

for name, group in plugin_map:
    prev_keys = set(data.keys())
    data = runner.get_data(data, name, group, args_map[name])

    if runner.adds_keys(name, group):
        new_keys = set(data.keys()) - prev_keys
        logger.log(new_keys)

python $PYTHONPATH/sofine/runner.py '--SF-s ystockquotelib --SF-g example --SF-a adds_keys'

curl -X POST -d '{}' --header "Content-Type:application/json" http://localhost:10000/SF-s/ystockquotelib/SF-g/example/SF-a/adds_keys

data_source = 'ystockquotelib'
data_source_group = 'example'
adds_keys = runner.adds_keys(data_source, data_source_group)

{"adds_keys": true|false}

Additional Convenience Methods

Plugins called from Python also expose two convenience methods that let you get a reference to the plugin’s module or to the plugin’s class.

data_source = 'google_search_results'
data_source_group = 'example'
plugin = runner.get_plugin(data_source, data_source_group)
schema = plugin.schema

data_source = 'google_search_results'
data_source_group = 'example'
mod = runner.get_plugin_module(data_source, data_source_group)
# The google plugin implements an additional helper method in the module that returns
# the list of attributes in each 'results' object it returns mapped to each key
child_shema = mod.get_child_schema()

Managing Data Retrieval Plugins

Managing data retrieval plugins is very simple. Pick a directory from which you want to call your plugins. Define the environment variable SOFINE_PLUGIN_PATH and assign to it the path to your plugin directory.

Plugins themselves are just Python modules fulfilling the requirements detailed in the section, “How Plugins Work and How to Write Them.”

Plugins cannot be deployed at the root of your plugin directory. Instead you must create one or more subdirectories and place plugins in them. Any plugin can live in any subdirectory. If you want, you can even place a plugin in more than one plugin directory. The plugin module name must match the plugin’s self.name attribute, and the plugin directory name must match the plugin’s self.group attribute.

This approach means you can manage your plugin directory without any dependencies on sofine. You can manage your plugins directory as their own code repo, and include unit tests or config files in the plugin directory, etc.

Managing Data Format Plugins

Pick a direcgory from which you want to call your plugins. Define the environment variable SOFINE_DATA_FORMAT_PLUGIN_PATH and assign it to the path of your plugin directory.

Unlike data retrieval plugins, data format plugins should be deployed directly in your plugin directory, not in a subdirectory.

Data format plugins are simply modules. By convention they should be named format_<FORMAT_NAME>.py, for example, format_json.py. This is optional, but provides a standard way to avoid name clashes with built-in or third-party modules named after a data format, such as the Python standard library json and csv modules.

Appendix: The Data Retrieval Algorithm

• The returned data set (let’s call it “data”) is always a JSON object of string keys mapped to object values.

• On every call in a sofine chain, add any new keys returned to data, and add all key attribute data returned to that key in data.

• All attributes mapped to a key are JSON objects which themselves consist of string keys mapped to legal JSON values.

So, formally, the result of a call to a sofine pipe is the union of all keys retrieved by all plugin calls, with each key mapped to the union of all attributes returned by all plugin calls for that key.

Developing With the sofine Code Base

All of the above documentation covers the very common case of using sofine as a library to manage and call your own plugins.

However, you might want to develop with sofine more directly. Perhaps you want to use pieces of the library for other purposes, or fork the library to add features, or even contribute!

In that case, you’ll want the developer documentation: http://marksweiss.github.io/sofine/