pandalone 0.3.4

Overview

At the most fundamental level, an “execution” or a “run” of any data-processing can be thought like that:

      .--------------.     _____________        .-------------.
     ;  DataTree    ;    |             |      ;   DataTree   ;
    ;--------------; ==> |  <cfunc_1>  | ==> ;--------------;
   ; /some/data   ;      |  <cfunc_2>  |    ; /some/data   ;
  ;  /some/other ;       |     ...     |   ;  /some/other ;
 ;   /foo/bar   ;        |_____________|  ;   /foo/bar   ;
'--------------'                         '--------------.

• The data-tree might come from json, hdf5, excel-workbooks, or plain dictionaries and lists. Its values are strings and numbers, numpy-lists, pandas or xray-datasets, etc.

• The component-functions must abide to the following simple signature:

  cfunc_do_something(pandelone, datatree)

  and must not return any value, just read and write into the data-tree.

• Here is a simple component-function:

  def cfunc_standardize(pandelone, datatree):
      pin, pon = pandelone.paths(),
      df = datatree.get(pin.A)
      df[pon.A.B_std] = df[pin.A.B] / df[pin.A.B].std()

• Notice the use of the relocatable-paths marked specifically as input or output.

• TODO: continue rough example in tutorial…

Quick-start

The program runs on Python-3.5+ and requires numpy, pandas and (optionally) win32 libraries along with their native backends. .. code-block:: bash

pip install pandalone ## Use –pre if version-string has a build-suffix.

Or in case you need the very latest from master branch :

pip install git+https://github.com/pandalone/pandalone.git

Or in to install in develop mode, with all dependencies needed for development, and with pre-commit hook for auto-formatting python-code with black, clone locally this project from the remote repo, and run:

pip install -e <pandalone-dr>[dev]
pre-commit install

Project files and folders

The files and folders of the project are listed below:

+--pandalone/       ## (package) Python-code
+--tests/           ## (package) Test-cases
+--doc/             ## Documentation folder
+--setup.py         ## (script) The entry point for `setuptools`, installing, testing, etc
+--requirements/    ## (txt-files) Various pip and conda dependencies.
+--README.rst
+--CHANGES.rst
+--AUTHORS.rst
+--CONTRIBUTING.rst
+--LICENSE.txt

GUI usage

For a quick-‘n-dirty method to explore the structure of the data-tree and run an experiment, just run:

$ pandalone gui

Excel usage

In Windows and OS X you may utilize the excellent xlwings library to use Excel files for providing input and output to the experiment.

To create the necessary template-files in your current-directory you should enter:

$ pandalone excel

You could type instead samp(pandalone excel {file_path}) to specify a different destination path.

[TBD]

Python usage

Example python REPL (Read-Eval-Print Loop) example-commands are given below that setup and run an experiment.

First run command(python) or command(ipython) and try to import the project to check its version:

code-block:

>>> import pandalone

>>> pandalone.__version__           ## Check version once more.
'0.3.4'

>>> pandalone.__file__              ## To check where it was installed.         # doctest: +SKIP
/usr/local/lib/site-package/pandalone-...

If everything works, create the data-tree to hold the input-data (strings and numbers). You assemble data-tree by the use of:

• sequences,

• dictionaries,

• class(pandas.DataFrame),

• class(pandas.Series), and

• URI-references to other data-trees.

[TBD]

Sources & Dependencies

To get involved with development, you need a POSIX environment to fully build it (Linux, OSX or Cygwin on Windows).

First you need to download the latest sources:

$ git clone https://github.com/pandalone/pandalone.git pandalone.git
$ cd pandalone.git

Then you can install all project’s dependencies in `development mode using the file(setup.py) script:

$ python setup.py --help                           ## Get help for this script.
Common commands: (see '--help-commands' for more)

  setup.py build      will build the package underneath 'build/'
  setup.py install    will install the package

Global options:
  --verbose (-v)      run verbosely (default)
  --quiet (-q)        run quietly (turns verbosity off)
  --dry-run (-n)      don't actually do anything
...

$ python setup.py develop                           ## Also installs dependencies into project's folder.
$ python setup.py build                             ## Check that the project indeed builds ok.

You should now run the test-cases to check that the sources are in good shape:

$ python setup.py test

Design

See architecture live-document.

Why another XXX? What about YYY?

These are the knowingly related python projects:

• OpenMDAO: It has influenced pandalone’s design. It is planned to interoperate by converting to and from it’s data-types. But it is Python-2 only and its architecture needs attending from programmers (no setup.py, no official test-cases).

• PyDSTool: It does not overlap, since it does not cover IO and dependencies of data. Also planned to interoperate with it (as soon as we have a better grasp of it :-). It has some issues with the documentation, but they are working on it.

• xray: Pandas for higher dimensions; data-trees should in principle work with “xray”.

• Blaze: NumPy and Pandas interface to Big Data; data-trees should in principle work with “blaze”.

• netCDF4: Hierarchical file-data-format similar to hdf5; a data-tree may derive in principle from “netCDF4 “.

• hdf5: Hierarchical file-data-format, supported natively by pandas; a data-tree may derive in principle from “netCDF4 “.

Which other projects/ideas have you reviewed when building this library?

• bubbles ETL: Processing-pipelines for (mostly) categorical data.

• Data-protocols:

  • JTSKit, A utility library for working with JSON Table Schema in Python.

  • Data Packages

• Celery: Execute distributed asynchronous tasks using message passing on a single or more worker servers using multiprocessing, Eventlet, or gevent.

• Fuzzywuzzy and Jellyfish: Fuzzy string matching in python. Use it for writting code that can read coarsely-known column-names.

• “Other’s people’s messy data (and how not to hate it)”, PyCon 2015(Canada) presentation by Mali Akmanalp.