cogeo-mosaic

Create mosaics of Cloud Optimized GeoTIFF based on mosaicJSON specification.

Read the official announcement https://medium.com/devseed/cog-talk-part-2-mosaics-bbbf474e66df

This python module provide a CLI to help create mosaicJSON.

Install (python >=3)

$ pip install pip -U
$ pip install cogeo-mosaic

# Or from source

$ pip install git+http://github.com/developmentseed/cogeo-mosaic

Notes:

• Starting with version 2.0, pygeos has replaced shapely and thus makes libgeos a requirement.
• pygeos hosted on pypi migth not compile on certain machine. This has been fixed in the master branch and can be installed with pip install git+https://github.com/pygeos/pygeos.git

CLI

$ cogeo-mosaic --help
Usage: cogeo-mosaic [OPTIONS] COMMAND [ARGS]...

  cogeo_mosaic cli.

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  create                Create mosaic definition from list of files
  create-from-features  Create mosaic definition from GeoJSON features or features collection
  footprint             Create geojson from list of files
  info                  Return info about the mosaic
  overview              [EXPERIMENTAL] Create a low resolution mosaic image from a MosaicJSON.
  update                Update a mosaic definition from list of files
  upload                Upload mosaic definition to backend

Create Mosaic definition

$ cogeo-mosaic create --help
Usage: cogeo-mosaic create [OPTIONS] [INPUT_FILES]

  Create mosaic definition file.

Options:
  -o, --output PATH       Output file name
  --minzoom INTEGER       An integer to overwrite the minimum zoom level derived from the COGs.
  --maxzoom INTEGER       An integer to overwrite the maximum zoom level derived from the COGs.
  --quadkey-zoom INTEGER  An integer to overwrite the quadkey zoom level used for keys in the MosaicJSON.
  --min-tile-cover FLOAT  Minimum % overlap
  --tile-cover-sort       Sort files by covering %
  --threads INTEGER       threads
  -q, --quiet             Remove progressbar and other non-error output.
  --help                  Show this message and exit.

[INPUT_FILES] must be a list of valid Cloud Optimized GeoTIFF.

$ cogeo-mosaic create list.txt -o mosaic.json

# or

$ cat list.txt | cogeo-mosaic create - | gzip > mosaic.json.gz

# or use backends like AWS S3 or DynamoDB

$ cogeo-mosaic create list.txt -o s3://my-bucket/my-key.json.gz

Example: create a mosaic from OAM

# Create Mosaic
$ curl https://api.openaerialmap.org/user/5d6a0d1a2103c90007707fa0 | jq -r '.results.images[] | .uuid' | cogeo-mosaic create - | gzip >  5d6a0d1a2103c90007707fa0.json.gz

# Create Footprint (optional)
$ curl https://api.openaerialmap.org/user/5d6a0d1a2103c90007707fa0 | jq -r '.results.images[] | .uuid' | cogeo-mosaic footprint | gist -p -f test.geojson

Create Mosaic definition from a GeoJSON features collection (e.g STAC)

This module is first design to create mosaicJSON from a set of COG urls but starting in version 3.0.0 we have added a CLI to be able to create mosaicJSON from GeoJSON features.

$ cogeo-mosaic create-from-features --help
Usage: cogeo-mosaic create-from-features [OPTIONS] FEATURES...

  Create mosaic definition file.

Options:
  -o, --output PATH       Output file name
  --minzoom INTEGER       Mosaic minimum zoom level.  [required]
  --maxzoom INTEGER       Mosaic maximum zoom level.  [required]
  --property TEXT         Define accessor property  [required]
  --quadkey-zoom INTEGER  An integer to overwrite the quadkey zoom level used for keys in the MosaicJSON.
  --min-tile-cover FLOAT  Minimum % overlap
  --tile-cover-sort       Sort files by covering %
  -q, --quiet             Remove progressbar and other non-error output.
  --help                  Show this message and exit.

Use it with STAC

curl https://earth-search.aws.element84.com/collections/landsat-8-l1/items | cogeo-mosaic create-from-features --minzoom 7 --maxzoom 12 --property "landsat:scene_id" --quiet | jq

{
  "mosaicjson": "0.0.2",
  "version": "1.0.0",
  "minzoom": 7,
  "maxzoom": 12,
  "quadkey_zoom": 7,
  "bounds": [16.142300225571994, -28.513088675819393, 67.21380296165974, 81.2067478836583],
  "center": [41.67805159361586, 26.346829603919453, 7],
  "tiles": {
    "1012123": [
      "LC81930022020114LGN00"
    ],
    ...
  }
}

Create Mosaic Overview [experimental]

The CLI provides an overview command to create low-resolution version of a mosaic. This is hightly experimental and might incure some cost if you are hosting mosaic on DynamoDB or COG files on S3. To create the overview, the overview method will fetch all the asset's overviews (COG internal overview) and construct one or multiple COG .

$ cogeo-mosaic overview s3://bucket/mymosaic.json

Image Order

By default the order of the dataset, either passed via the CLI or in the API, defines the order of the quadkey's assets.

from cogeo_mosaic.mosaic import MosaicJSON

# list of COG
dataset = ["1.tif", "2.tif"]
mosaic_definition = MosaicJSON.from_urls(dataset)

print(mosaic_definition.tiles)
> {"tile": {"0": ["cog1.tif", "2.tif"]}}

API

Mosaic Storage Backends

Starting in version 3.0.0, we introduced specific backend storage for:

• File (default, file:///)

• HTTP/HTTPS (http://, https://)

• AWS S3 (s3://)

• AWS DynamoDB (dynamodb://{region}/{table_name}). If region is not passed, it reads the value of the AWS_REGION environment variable. If that environment variable does not exist, it falls back to us-east-1. If you choose not to pass a region, you still need three / before the table name, like so dynamodb:///{table_name}.

• STAC (stac+:https://). Based on SpatioTemporal Asset Catalog API.

To ease the usage we added a helper function to use the right backend based on the uri schema: cogeo_mosaic.backends.MosaicBackend

from cogeo_mosaic.backends import MosaicBackend

with MosaicBackend("s3://mybucket/amosaic.json") as mosaic:
    assert isinstance(mosaic, cogeo_mosaic.backends.s3.S3Backend)

with MosaicBackend("https://mosaic.com/amosaic.json.gz") as mosaic:
    assert isinstance(mosaic, cogeo_mosaic.backends.http.HttpBackend)

with MosaicBackend("dynamodb://us-east-1/amosaic") as mosaic:
    assert isinstance(mosaic, cogeo_mosaic.backends.dynamodb.DynamoDBBackend)

with MosaicBackend("file:///amosaic.json.gz") as mosaic:
    assert isinstance(mosaic, cogeo_mosaic.backends.file.FileBackend)

with MosaicBackend("stac+https://my-stac.api/search", {"collections": ["satellite"]}) as mosaic:
    assert isinstance(mosaic, cogeo_mosaic.backends.stac.STACBackend)

with MosaicBackend("amosaic.json.gz") as mosaic:
    assert isinstance(mosaic, cogeo_mosaic.backends.file.FileBackend)

Properties and Methods

Each MosaicBackends (file, http, s3, dynamodb and stac) extend rio-tiler BaseReader and thus derives the same minimal methods/properties

from cogeo_mosaic.backends import MosaicBackend

# Read
with MosaicBackend("s3://mybucket/amosaic.json") as mosaic:
    mosaic.mosaic_def                  # property - MosaicJSON document, wrapped in a Pydantic Model
    mosaic.metadata                    # property - Return mosaic metadata
    mosaic.mosaicid                    # property - Return sha224 id from the mosaicjson doc
    mosaic.quadkey_zoom                # property - Return Quadkey zoom of the mosaic

    mosaic.minzoom                     # property - Mosaic minzoom
    mosaic.maxzoom                     # property - Mosaic maxzoom
    mosaic.bounds                      # property - Mosaic bounds
    mosaic.spatial_info                # property - zooms and bounds info

    mosaic.info                        # method -  spatial_info, list of quadkeys and mosaic name

    mosaic.assets_for_tile(1,2,3)      # method - Find assets for a specific mercator tile
    mosaic.assets_for_point(lng, lat)  # method - Find assets for a specific point

    mosaic.tile(1,2,3)                 # method - Create mosaic tile
    mosaic.point(lng, lat)             # method - Read point value from multiple assets
    mosaic.write()                     # method - Write the mosaicjson to the given location
    mosaic.update([features])          # method - Update the mosaicjson data with a list of features

Read and Get assets list

# MosaicBackend is the top level backend and will distribute to the
# correct backend by checking the path/url schema.
from cogeo_mosaic.backends import MosaicBackend

with MosaicBackend("s3://mybucket/amosaic.json") as mosaic:
    assets: List = mosaic.assets_for_tile(1, 2, 3) # get assets for mercantile.Tile(1, 2, 3)

Read Tile Data (mosaic tile)

# MosaicBackend is the top level backend and will distribute to the
# correct backend by checking the path/url schema.
from cogeo_mosaic.backends import MosaicBackend

with MosaicBackend("s3://mybucket/amosaic.json") as mosaic:
    tile, mask = mosaic.tile(1, 2, 3)

Write

from cogeo_mosaic.utils import create_mosaic
from cogeo_mosaic.backends import MosaicBackend

mosaicdata = create_mosaic(["1.tif", "2.tif"])

with MosaicBackend("s3://mybucket/amosaic.json", mosaic_def=mosaicdata) as mosaic:
    mosaic.write() # trigger upload to S3

Update

from cogeo_mosaic.utils import get_footprints
from cogeo_mosaic.backends import MosaicBackend

with MosaicBackend("s3://mybucket/amosaic.json") as mosaic:
    features = get_footprints(["3.tif"]) # Get footprint
    mosaic.update(features) # Update mosaicJSON and upload to S3

In Memory

from cogeo_mosaic.utils import create_mosaic
from cogeo_mosaic.backends import MosaicBackend

mosaicdata = create_mosaic(["1.tif", "2.tif"])

with MosaicBackend(None, mosaic_def=mosaicdata) as mosaic:
    tile, mask = mosaic.tile(1, 2, 3)

STAC: SpatioTemporal Asset Catalog

The STACBackend is purely dynamic, meaning it's not used to read or write a file. This backend will POST to the input url looking for STAC items which will then be used to create the mosaicJSON.

see /docs/STAC_backend.md for more info.

COGReader / STACReader

The mosaic object has .tile and .point methods to access the data for a specific mercator tile or point.

Because a MosaicJSON can host different assets type, a reader option is available. Set by default to rio_tiler.io.COGReader, or to rio_tiler.io.STACReader for the STACBackend, the reader should know how to read the assets to either create mosaic tile or read points value.

from cogeo_mosaic.mosaic import MosaicJSON
from cogeo_mosaic.backends import MosaicBackend

dataset = ["1.tif", "2.tif"]
mosaic_definition = MosaicJSON.from_urls(dataset)

# Create a mosaic object in memory
with MosaicBackend(None, mosaid_def=mosaic_definition, reader=COGReader) as mosaic:
    tile, mask = mosaic.tile(1, 1, 1)

# By default the STACbackend will store the Item url as assets, but STACReader (default reader) will know how to read them.
with MosaicBackend(
  "stac+https://my-stac.api/search",
  {"collections": ["satellite"]},
  7,  # minzoom
  12, # maxzoom
) as mosaic:
    tile, mask = mosaic.tile(1, 1, 1, assets="red")

Let's use a custom accessor to save some specific assets url in the mosaic

# accessor to return the url for the `visual` asset (COG)
def accessor(item):
    return feature["assets"]["visual"]["href"]

# The accessor will set the mosaic assets as a list of COG url so we can use the COGReader instead of the STACReader
with MosaicBackend(
  "stac+https://my-stac.api/search",
  {"collections": ["satellite"]},
  7,  # minzoom
  12, # maxzoom
  reader=COGReader,
  backend_options={"accessor": accessor},
) as mosaic:
    tile, mask = mosaic.tile(1, 1, 1)

Associated Modules

• cogeo-mosaic-tiler: A serverless stack to serve and vizualized tiles from Cloud Optimized GeoTIFF mosaic.

• cogeo-mosaic-viewer: A local Cloud Optimized GeoTIFF mosaic viewer based on rio-viz.

Contribution & Development

Issues and pull requests are more than welcome.

Dev install & Pull-Request

$ git clone http://github.com/developmentseed/cogeo-mosaic.git
$ cd cogeo-mosaic
$ pip install -e .[dev]

Python >=3.7 only

This repo is set to use pre-commit to run isort, flake8, pydocstring, black ("uncompromising Python code formatter") and mypy when committing new code.

$ pre-commit install

$ git add .

$ git commit -m'my change'
isort....................................................................Passed
black....................................................................Passed
Flake8...................................................................Passed
Verifying PEP257 Compliance..............................................Passed
mypy.....................................................................Passed

$ git push origin

About

Created by Development Seed