SDAP experiment with object storage and other technologies (ray.io, xarray...)

Features

• Applies simple algorithms on earth observation datasets

• Read Cloud Optimized GeoTiff on S3 bucket

• Manage S3 bucket with public access or with access keys

• Manage datasets in any CRS (WGS84, UTM, ...)

• Manage multi-band datasets

• Algorithms managed:

  • Spatial Mean: on a x,y,t range averages on x,y dimensions
  • EVI: linear combinations of multi-band spectra

Prerequisites

• python 3.9
• datasets on s3

Configure application and test

The collection configuration should follow the template found on src/sdap/test/data_access/collection-config.yaml

If some of the data you need to access on S3 requires authentication, use a secrets files as the one provided in example src/sdap/test/data_access/secrets.yaml.

You have one secret configuration (e.g. AWS credential) per collection.

Install and run tests:

pip install -e '.[dev]'
python setup.py test &> out.log &

Run

As a command line (suitable for serverless deployments)

Use command line

    sdap-cmd --help

For example with EVI operator:

    sdap-cmd --conf ./src/sdap/test/data_access/collection-config.yaml \
         --collection hls \
         --x-range -71.24 -71.23 \
         --y-range 42.40 42.42 \
         --time-range 2017-01-01T00:00:00.000000+00:00 2017-06-01T00:00:00.000000+00:00 \
         --operator-name EVI \
         --operator-args '[[0,0,-2.5,2.5,0,0,0],[0,0,2.4,1,0,0,1]]' \
         --plot

With SpatialMean operator:

    sdap-cmd --conf ./src/sdap/test/data_access/collection-config.yaml \
         --secrets my_secrets.yaml \
         --collection hls \
         --bbox -71.24 42.40 -71.23 42.42 \
         --time-range 2017-01-01T00:00:00.000000+00:00 2017-06-01T00:00:00.000000+00:00 \
         --operator-name SpatialMean

    sdap-cmd --conf ./src/sdap/test/data_access/collection-config.yaml \
         --collection maiac \
         --bbox -71.25 42.39 -71.23 42.42 \
         --time-range 2019-01-01T00:00:00.000000+00:00 2019-02-01T00:00:00.000000+00:00 \
         --operator-name SpatialMean

As a service

Starts a web server with an API:

sdap-serv  --conf src/sdap/test/data_access/collection-config.yaml --secrets my_secrets.yaml

Test the service, you can use the jupyter notebook(s) provided on src/sdap/notebook

You can run it locally by running:

pip install -e '.[client]'
jupyter-lab

Then select the notebook in the jupyuter web console and execute it, step by step.

Resource allocation

The resources needed to run the application can be estimated as follows:

• maximum size of 1 tile for each worker = T = 64 * x * y * t * obs dim, for example 643660366016 = 5Gb
• maximum size of the user result returned to the user = R = 64xytobs dims
• number of parallels workers: n

Total RAM = T*n + R

Number of CPU = n

For developers

pip install -e '.[dev]'

Do you work, aad unit tests

Validate

python setup.py test

Package

Upgrade version in `src/sdap/VERSION.txt`

python setup.py sdist bdist_wheel

Create and publish the docker image

docker build . -t tloubrieu/sdap-os:0.1.1
docker push tloubrieu/sdap-os:0.1.1   

Create the kubernetes namespace:

kubectl create namespace sdap-os

Deploy the ray.io operator:

(See instructions on https://docs.ray.io/en/latest/cluster/kubernetes.html#ray-helm)

git clone https://github.com/ray-project/ray.git
cd ray/deploy/charts 
helm -n ray install sdap-cluster --create-namespace ./ray

Deploy the helm chart

helm install sdap-os ./helm/sdap-os --dependency-update -n sdap-os

Stop the service:

helm delete sdap-os -n sdap-os

On a local deployment with single node, the node might be tainted to prevent pods from being deployed on it. In this case you can run something like:

kubectl taint node docker-desktop node.kubernetes.io/disk-pressure:NoSchedule-

(see https://stackoverflow.com/questions/62991596/1-nodes-had-taints-that-the-pod-didnt-tolerate-in-kubernetes-cluster for more details)

Use with local minio

Set minio server (not used in the development yet)

Standalone minio:

Server install:

brew install minio/stable/minio

Start the server

export MINIO_ROOT_USER=myminio
export MINIO_ROOT_PASSWORD=minio123AB

minio server /Users/loubrieu/Documents/sdap/minio --address localhost:9000 --console-address localhost:19000

Client

brew install minio/stable/mc
mc --help

Create server alias:

mc alias set minio http://localhost:9000 myminio minio123AB

Create a bucket:

mc mb hls

Copy test datasets

Create a bucket:

mc mb hls

Copy some files:

mc cp data/hls/s1_output_latlon_HLS_S30_T18TYN_2019263_cog.tif minio/hls
mc cp data/hls/s1_output_latlon_HLS_S30_T18TYN_2019260_cog.tif minio/hls
mc cp data/hls/s1_output_latlon_HLS_S30_T18TYN_2019253_cog.tif minio/hls