aghast

Aghast is a histogramming library that does not fill histograms and does not plot them. Its role is behind the scenes, to provide better communication between histogramming libraries.

Specifically, it is a structured representation of aggregated, histogram-like statistics as sharable "ghasts." It has all of the "bells and whistles" often associated with plain histograms, such as number of entries, unbinned mean and standard deviation, bin errors, associated fit functions, profile plots, and even simple ntuples (needed for unbinned fits or machine learning applications). ROOT has all of these features; Numpy has none of them.

The purpose of aghast is to be an intermediate when converting ROOT histograms into Numpy, or vice-versa, or both of these into Boost.Histogram, Physt, Pandas, etc. Without an intermediate representation, converting between N libraries (to get the advantages of all) would equire N(N ‒ 1)/2 conversion routines; with an intermediate representation, we only need N, and the mapping of feature to feature can be made explicit in terms of a common language.

Furthermore, aghast is a Flatbuffers schema, so it can be deciphered in many languages, with lazy, random-access, and uses a small amount of memory. A collection of histograms, functions, and ntuples can be shared among processes as shared memory, used in remote procedure calls, processed incrementally in a memory-mapped file, or saved in files with future-proof schema evolution.

Installation from packages

Not on PyPI or conda yet. Refer to manual installation.

Manual installation

After you git-clone this GitHub repository and ensure that numpy is installed, somehow:

pip install "flatbuffers>=1.8.0"  # for the flatbuffers Python runtime with Numpy interface
cd python                         # only implementation so far is in Python
python setup.py install           # if you want to use it outside of this directory

Now you should be able to import aghast or from aghast import * in Python.

If you need to change flatbuffers/aghast.fbs, you'll need to additionally:

1. Get flatc to generate Python sources from flatbuffers/aghast.fbs. I use conda install -c conda-forge flatbuffers. (The flatc executable is not included in the pip flatbuffers package, and the Python runtime is not included in the conda flatbuffers package. They're disjoint.)
2. In the python directory, run ./generate_flatbuffers.py (which calls flatc and does some post-processing).

Every time you change flatbuffers/aghast.fbs, re-run ./generate_flatbuffers.py.

Documentation

Suite of examples as a Jupyter notebook:

• in GitHub
• on Binder

Full specification:

• Introduction
• Data types
  • Collection
  • Histogram
  • Axis
  • IntegerBinning
  • RegularBinning
  • RealInterval
  • RealOverflow
  • HexagonalBinning
  • EdgesBinning
  • IrregularBinning
  • CategoryBinning
  • SparseRegularBinning
  • FractionBinning
  • PredicateBinning
  • VariationBinning
  • Variation
  • Assignment
  • UnweightedCounts
  • WeightedCounts
  • InterpretedInlineBuffer
  • InterpretedInlineInt64Buffer
  • InterpretedInlineFloat64Buffer
  • InterpretedExternalBuffer
  • Profile
  • Statistics
  • Moments
  • Quantiles
  • Modes
  • Extremes
  • StatisticFilter
  • Covariance
  • ParameterizedFunction
  • Parameter
  • EvaluatedFunction
  • BinnedEvaluatedFunction
  • Ntuple
  • Column
  • NtupleInstance
  • Chunk
  • ColumnChunk
  • Page
  • RawInlineBuffer
  • RawExternalBuffer
  • Metadata
  • Decoration