Skip to main content

Demo of usage of object storage for SDAP

Project description

SDAP experiment with object storage and other technologies (ray.io, xarray...)

Features

  • Applies simple algorithms on earth observation datasets

  • Read Cloud Optimized GeoTiff on S3 bucket

  • Manage S3 bucket with public access or with access keys

  • Manage datasets in any CRS (WGS84, UTM, ...)

  • Manage multi-band datasets

  • Algorithms managed:

    • Spatial Mean: on a x,y,t range averages on x,y dimensions
    • EVI: linear combinations of multi-band spectra

Prerequisites

  • python 3.9
  • datasets on s3

Configure application and test

The collection configuration should follow the template found on src/sdap/test/data_access/collection-config.yaml

If some of the data you need to access on S3 requires authentication, use a secrets files as the one provided in example src/sdap/test/data_access/secrets.yaml.

You have one secret configuration (e.g. AWS credential) per collection.

Install and run tests:

pip install -e '.[dev]'
python setup.py test &> out.log &

Run

As a command line (suitable for serverless deployments)

Use command line

    sdap-cmd --help

For example with EVI operator:

    sdap-cmd --conf ./src/sdap/test/data_access/collection-config.yaml \
         --collection hls \
         --x-range -71.24 -71.23 \
         --y-range 42.40 42.42 \
         --time-range 2017-01-01T00:00:00.000000+00:00 2017-06-01T00:00:00.000000+00:00 \
         --operator-name EVI \
         --operator-args '[[0,0,-2.5,2.5,0,0,0],[0,0,2.4,1,0,0,1]]' \
         --plot

With SpatialMean operator:

    sdap-cmd --conf ./src/sdap/test/data_access/collection-config.yaml \
         --secrets my_secrets.yaml \
         --collection hls \
         --bbox -71.24 42.40 -71.23 42.42 \
         --time-range 2017-01-01T00:00:00.000000+00:00 2017-06-01T00:00:00.000000+00:00 \
         --operator-name SpatialMean

    sdap-cmd --conf ./src/sdap/test/data_access/collection-config.yaml \
         --collection maiac \
         --bbox -71.25 42.39 -71.23 42.42 \
         --time-range 2019-01-01T00:00:00.000000+00:00 2019-02-01T00:00:00.000000+00:00 \
         --operator-name SpatialMean

As a service

Starts a web server with an API:

sdap-serv  --conf src/sdap/test/data_access/collection-config.yaml --secrets my_secrets.yaml

Test the service, you can use the jupyter notebook(s) provided on src/sdap/notebook

You can run it locally by running:

pip install -e '.[client]'
jupyter-lab

Then select the notebook in the jupyuter web console and execute it, step by step.

Resource allocation

The resources needed to run the application can be estimated as follows:

  • maximum size of 1 tile for each worker = T = 64 * x * y * t * obs dim, for example 643660366016 = 5Gb
  • maximum size of the user result returned to the user = R = 64xytobs dims
  • number of parallels workers: n

Total RAM = T*n + R

Number of CPU = n

For developers

pip install -e '.[dev]'

Do you work, aad unit tests

Validate

python setup.py test

Package

Upgrade version in `src/sdap/VERSION.txt`

python setup.py sdist bdist_wheel

Create and publish the docker image

docker build . -t tloubrieu/sdap-os:0.1.1
docker push tloubrieu/sdap-os:0.1.1   

Create the kubernetes namespace:

kubectl create namespace sdap-os

Deploy the ray.io operator:

(See instructions on https://docs.ray.io/en/latest/cluster/kubernetes.html#ray-helm)

git clone https://github.com/ray-project/ray.git
cd ray/deploy/charts 
helm -n ray install sdap-cluster --create-namespace ./ray

Deploy the helm chart

helm install sdap-os ./helm/sdap-os --dependency-update -n sdap-os

Stop the service:

helm delete sdap-os -n sdap-os

On a local deployment with single node, the node might be tainted to prevent pods from being deployed on it. In this case you can run something like:

kubectl taint node docker-desktop node.kubernetes.io/disk-pressure:NoSchedule-

(see https://stackoverflow.com/questions/62991596/1-nodes-had-taints-that-the-pod-didnt-tolerate-in-kubernetes-cluster for more details)

Use with local minio

Set minio server (not used in the development yet)

Standalone minio:

Server install:

brew install minio/stable/minio

Start the server

export MINIO_ROOT_USER=myminio
export MINIO_ROOT_PASSWORD=minio123AB

minio server /Users/loubrieu/Documents/sdap/minio --address localhost:9000 --console-address localhost:19000

Client

brew install minio/stable/mc
mc --help

Create server alias:

mc alias set minio http://localhost:9000 myminio minio123AB

Create a bucket:

mc mb hls

Copy test datasets

Create a bucket:

mc mb hls

Copy some files:

mc cp data/hls/s1_output_latlon_HLS_S30_T18TYN_2019263_cog.tif minio/hls
mc cp data/hls/s1_output_latlon_HLS_S30_T18TYN_2019260_cog.tif minio/hls
mc cp data/hls/s1_output_latlon_HLS_S30_T18TYN_2019253_cog.tif minio/hls

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

sdap-os-0.1.2.tar.gz (25.5 kB view hashes)

Uploaded Source

Built Distribution

sdap_os-0.1.2-py3-none-any.whl (37.9 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page