Skip to main content

Windowed multiprocessing wrapper for rasterio

Project description

rio-mucho
=========

Parallel processing wrapper for rasterio

|PyPI| |Build Status| |Coverage Status|

Install
-------

From pypi:

``pip install rio-mucho``

From github (usually for a branch / dev):

``pip install pip install git+ssh://git@github.com/mapbox/rio-mucho.git@<branch>#egg=riomucho``

Development:

::

git clone git@github.com:mapbox/rio-mucho.git
cd rio-mucho
pip install -e .

Usage
-----

.. code:: python

with riomucho.RioMucho([{inputs}], {output}, {run function},
windows={windows},
global_args={global arguments},
options={options to write}) as rios:

rios.run({processes})

Arguments
~~~~~~~~~

``inputs``
^^^^^^^^^^

An list of file paths to open and read.

``output``
^^^^^^^^^^

What file to write to.

``run_function``
^^^^^^^^^^^^^^^^

A function to be applied to each window chunk. This should have input
arguments of:

1. A data input, which can be one of:

- A list of numpy arrays of shape (x,y,z), one for each file as
specified in input file list ``mode="simple_read" [default]``
- A numpy array of shape ({*n* input files x *n* band count}, {window
rows}, {window cols}) ``mode=array_read"``
- A list of open sources for reading ``mode="manual_read"``

2. A ``rasterio`` window tuple
3. A ``rasterio`` window index (``ij``)
4. A global arguments object that you can use to pass in global
arguments

This should return:

1. An output array of ({count}, {window rows}, {window cols}) shape, and
of the correct data type for writing

.. code:: python

def basic_run({data}, {window}, {ij}, {global args}):
## do something
return {out}

Keyword arguments
~~~~~~~~~~~~~~~~~

``windows={windows}``
^^^^^^^^^^^^^^^^^^^^^

A list of ``rasterio`` (window, ij) tuples to operate on.
``[Default = src[0].block_windows()]``

``global_args={global arguments}``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Since this is working in parallel, any other objects / values that you
want to be accessible in the ``run_function``. ``[Default = {}]``

.. code:: python

global_args = {
'divide_value': 2
}

``options={keyword args}``
^^^^^^^^^^^^^^^^^^^^^^^^^^

The options to pass to the writing output. ``[Default = srcs[0].meta``

Example
-------

.. code:: python

import riomucho, rasterio, numpy

def basic_run(data, window, ij, g_args):
## do something
out = np.array(
[d[0] /= global_args['divide'] for d in data]
)
return out

# get windows from an input
with rasterio.open('/tmp/test_1.tif') as src:
## grabbing the windows as an example. Default behavior is identical.
windows = [[window, ij] for ij, window in src.block_windows()]
options = src.meta
# since we are only writing to 2 bands
options.update(count=2)

global_args = {
'divide': 2
}

processes = 4

# run it
with riomucho.RioMucho(['input1.tif','input2.tif'], 'output.tif', basic_run,
windows=windows,
global_args=global_args,
options=options) as rm:

rm.run(processes)

Utility functions
-----------------

\`riomucho.utils.array\_stack([array, array, array,...])
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Given a list of ({depth}, {rows}, {cols}) numpy arrays, stack into a
single (l{list length \* each image depth}, {rows}, {cols}) array. This
is useful for handling variation between ``rgb`` inputs of a single
file, or separate files for each.

One RGB file
^^^^^^^^^^^^

.. code:: python

files = ['rgb.tif']
open_files = [rasterio.open(f) for f in files]
rgb = `riomucho.utils.array_stack([src.read() for src in open_files])

Separate RGB files
^^^^^^^^^^^^^^^^^^

.. code:: python

files = ['r.tif', 'g.tif', 'b.tif']
open_files = [rasterio.open(f) for f in files]
rgb = `riomucho.utils.array_stack([src.read() for src in open_files])

.. |PyPI| image:: https://img.shields.io/pypi/v/rio-mucho.svg?maxAge=2592000?style=plastic
:target:
.. |Build Status| image:: https://travis-ci.org/mapbox/rio-mucho.svg?branch=master
:target: https://travis-ci.org/mapbox/rio-mucho
.. |Coverage Status| image:: https://coveralls.io/repos/mapbox/rio-mucho/badge.svg?branch=master&service=github
:target: https://coveralls.io/github/mapbox/rio-mucho?branch=master


Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

rio-mucho-1.0.0.tar.gz (5.8 kB view details)

Uploaded Source

Built Distributions

rio_mucho-1.0.0-py3-none-any.whl (5.8 kB view details)

Uploaded Python 3

rio_mucho-1.0.0-py2-none-any.whl (5.8 kB view details)

Uploaded Python 2

File details

Details for the file rio-mucho-1.0.0.tar.gz.

File metadata

  • Download URL: rio-mucho-1.0.0.tar.gz
  • Upload date:
  • Size: 5.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for rio-mucho-1.0.0.tar.gz
Algorithm Hash digest
SHA256 606f662e4cc0c7efb98dd7a1eb4df9d93022a305d187b6c02574c4542593b590
MD5 4caee5bc4f42bcd5bb459027840d29d0
BLAKE2b-256 022389ef962469c7942e26c3c5e904b209997b7313dea01c99ed6fa8607e077b

See more details on using hashes here.

File details

Details for the file rio_mucho-1.0.0-py3-none-any.whl.

File metadata

File hashes

Hashes for rio_mucho-1.0.0-py3-none-any.whl
Algorithm Hash digest
SHA256 4677aca0613b35b1eb6dcdf314eafc23b707c42e54e9af89a87c86df5ae1ba6c
MD5 5288688e5b47bedb88e584fc0c57d0db
BLAKE2b-256 8abae9a23efc6a8ffe6b2340c9f1040cd26a730754c75a58061c9302c66156fa

See more details on using hashes here.

File details

Details for the file rio_mucho-1.0.0-py2-none-any.whl.

File metadata

File hashes

Hashes for rio_mucho-1.0.0-py2-none-any.whl
Algorithm Hash digest
SHA256 156eb911521969e4a795ab438e02c9ac5bf5b441806fa78e556f0d1ea113a27c
MD5 aea8b22bca13ab4ab3b882f169809843
BLAKE2b-256 bbfb827facf90243f739f79bf670c6b1555a3739054c669231a6ea13da8b1ba6

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page