Skip to main content

Parallel processing wrapper for rasterio

Project description

rio-mucho

Parallel processing wrapper for rasterio

Build Status

Usage

  1. Define a function to be applied to each window chunk. This should have input arguments of:

  • A list of numpy arrays (one for each file as specified in input file list) of shape ({bands}, {window rows}, {window cols})

  • A rasterio window tuple

  • A rasterio window index (ij)

  • A global arguments object that you can use to pass in global arguments

def basic_run(data, window, ij, g_args):
    return data[0]
  1. Alternatively, for more flexibility, you can use a “manual read” where you read each raster in this function. This is useful if you want to read / write different window sizes (eg for pansharpening, or buffered window reading). Here, instead of a list of arrays, the function is passed an array of rasters open for reading.

def basic_run(open_files, window, ij, g_args):
    return numpy.array([f.read(window=window)[0] for f in open_files]) / g_args['divide']

For both of these, an array of identical shape to the destination window should be returned.

  1. To run, make some windows, get or make some keyword args for writing, and pass these and the above function into riomucho: ```python import riomucho, rasterio, numpy

get windows from an input

with rasterio.open(‘/tmp/test_1.tif’) as src: windows = [[window, ij] for ij, window in src.block_windows()] kwargs = src.meta # since we are only writing to 2 bands kwargs.update(count=2)

global_args = { ‘divide’: 2 }

processes = 4

run it

with riomucho.RioMucho([‘input1.tif’,’input2, input2.tif’], ‘output.tif’, basic_run, windows=windows, global_args=global_args, kwargs=kwargs) as rm:

rm.run(processes)

``` - If no windows are specified, rio-mucho uses the block windows of the first input raster - If no kwargs are specified, rio-mucho uses the kwargs of the first input dataset to write to output - If no global args are specified, an empty object is passed.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

rio-mucho-0.0.1dev.tar.gz (3.2 kB view details)

Uploaded Source

File details

Details for the file rio-mucho-0.0.1dev.tar.gz.

File metadata

File hashes

Hashes for rio-mucho-0.0.1dev.tar.gz
Algorithm Hash digest
SHA256 6fd16a96b146a106ee0d537964a441e84bdeb141257e303cab23912d9e328d71
MD5 d76f7fa685efd6e09092e56672d58112
BLAKE2b-256 7faf0ca8e964dbf756ac9f8246eb84cb3f99d8b0f1068ea9db29bc130f1d18a5

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page