hdrhistogram 0.1.2

Python API

Users of this library can generally play 2 roles (often both):

• histogram provisioning (record values)

• histogram query

A histogram instance can be created using the HdrHistogram class and specifying the minimum and maximum trackable value and the number of precision digits. For example to create a histogram that can count values in the [1..3600000] range and 1% precision (this could be for example to track latencies in the range [1 msec..1 hour]):

histogram = HdrHistogram(1, 60 * 60 * 1000, 2)

Once created it is easy to add values to it:

histogram.record_value(latency)

If the code that generates the values is subject to Coordinated Omission, use the corrected version of that method (example when the expected interval is 10 msec):

histogram.record_corrcted_value(latency, 10)

At any time, the histogram can be queried to return any property, such as getting the total number of values recorded or the value at a given percentile:

count = histogram.get_total_count()
value = histogram.get_value_at_percentile(99.9)

Recorded values can be iterated over using the recorded iterator:

for item in histogram.get_recorded_iterator():
    print('value=%f count=%d percentile=%f' %
            item.count_added_in_this_iter_step,
            item.value_iterated_to,
            item.percentile)

An encoded/compressed histogram can be generated by calling the compress method:

encoded_histogram = histogram.encode()

And on reception, a compressed histogram can be decoded from the encoded string:

decoded_histogram = HdrHistogram.decode(encoded_histogram)
count = decoded_histogram.get_total_count()

In the case of aggregation, the decode_and_add method can be used:

aggregation_histogram.decode_and_add(encoded_histogram)

For additional help on how to use the API:

• browse through the python code and check the API documentation in the comment section for each method (where available)

• the best documentation is by looking at the test code under the test directory

The test code (https://github.com/ahothan/hdrhistogram/blob/master/test/test_hdrhistogram.py) pretty much covers every API.

Acknowledgements

The python code was directly ported from the original HDR Histogram Java and C libraries:

• https://github.com/HdrHistogram/HdrHistogram.git

• https://github.com/HdrHistogram/HdrHistogram_c.git

Installation

Pre-requisites:

Make sure you have python 2.7 and pip installed

pip install hdrhistogram

pip install tox
# cd to the proper location to clone the repository
git clone https://github.com/ahothan/hdrhistogram.git
cd hdrhistogram

$ tox
GLOB sdist-make: /openstack/pyhdr/hdrhistogram/setup.py
py27 inst-nodeps: /openstack/pyhdr/hdrhistogram/.tox/dist/hdrhistogram-0.1.0.zip
py27 installed: flake8==2.4.1,hdrhistogram==0.1.0,mccabe==0.3.1,pep8==1.5.7,py==1.4.30,pyflakes==0.8.1,pytest==2.7.2,wsgiref==0.1.2
py27 runtests: PYTHONHASHSEED='40561919'
py27 runtests: commands[0] | py.test -q -s --basetemp=/openstack/pyhdr/hdrhistogram/.tox/py27/tmp

............................
28 passed in 6.40 seconds

pep8 inst-nodeps: /openstack/pyhdr/hdrhistogram/.tox/dist/hdrhistogram-0.1.0.zip
pep8 installed: flake8==2.4.1,hdr-histogram==0.1,hdrhistogram==0.1.0,mccabe==0.3.1,pep8==1.5.7,py==1.4.30,pyflakes==0.8.1,pytest==2.7.2,wsgiref==0.1.2
pep8 runtests: PYTHONHASHSEED='40561919'
pep8 runtests: commands[0] | flake8 hdrh test
________________________________________________________________________ summary _________________________________________________________________________
  py27: commands succeeded
  pep8: commands succeeded
  congratulations :)
$

Performance

Although not nearly as fast as the Java and C version, the python version provides adequate performance for most uses considering it is pure python code. Histogram value recording has the same cost characteristics than the Java version since it is a direct port (fixed cost for CPU and reduced memory usage). Encoding and decoding is relatively fast thanks to the use of native compression library (using zlib).

Aggregation of Distributed Histograms

Aggregation of multiple histograms into 1 is useful in cases where tools that generate these individual histograms have to run in a distributed way in order to scale sufficiently. As an example, the wrk2 tool (https://github.com/giltene/wrk2.git) is a great tool for measuring the latency of HTTP requests with a large number of connections. Although this tool can support thousands of connections per process, some setups require massive scale in the order of hundreds of thousands of connections which require running a large number of instances of wrk processes, possibly on a large number of servers. Given that each instance of wrk can generate a separate histogram, assessing the scale of the entire system requires aggregating all these histograms into 1 in a way that does not impact the accuracy of the results. So there are 2 problems to solve:

• find a way to properly aggregate multiple histograms without losing any detail

• find a way to transport all these histograms into a central place

This library provides a solution for the aggregation part of the problem:

• reuse the HDR histogram compression format version 1 to encode and compress a complete histogram that can be sent over the wire to the aggregator

• provide python APIs to easily and efficiently:

  • compress an histogram instance into a transportable string

  • decompress a compressed histogram and add it to an existing histogram

Refer to the unit test code (test/test_hdrhistogram.py) to see how these APIs can be used.

Limitations and Caveats

The latest features and bug fixes of the original HDR histogram library may not be available in this python port. List of notable features/APIs not implemented:

• concurrency support (AtomicHistogram, ConcurrentHistogram…)

• DoubleHistogram

• histogram auto-resize

• recorder function

Dependencies

The only dependency (outside of using pytest and tox for the unit testing) is the small pbr python package which takes care of the versioning (among other things).

Licensing

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Contribution

External contribution and forks are welcome.

Changes can be contributed back using preferably GerritHub (https://review.gerrithub.io/#/q/project:ahothan/hdrhistogram)

GitHub pull requests can also be considered.

Links

• Source: https://github.com/ahothan/hdrhistogram.git