Demo of usage of object storage for SDAP
Project description
SDAP experiment with object storage and other technologies (ray.io, xarray...)
Features
-
Applies simple algorithms on earth observation datasets
-
Read Cloud Optimized GeoTiff on S3 bucket
-
Manage S3 bucket with public access or with access keys
-
Manage datasets in any CRS (WGS84, UTM, ...)
-
Manage multi-band datasets
-
Algorithms managed:
- Spatial Mean: on a x,y,t range averages on x,y dimensions
- EVI: linear combinations of multi-band spectra
Prerequisites
- python 3.9
- datasets on s3
Configure application and test
The collection configuration should follow the template found on src/sdap/test/data_access/collection-config.yaml
If some of the data you need to access on S3 requires authentication, use a secrets files as the one provided in example src/sdap/test/data_access/secrets.yaml
.
You have one secret configuration (e.g. AWS credential) per collection.
Install and run tests:
pip install -e '.[dev]'
python setup.py test &> out.log &
Run
As a command line (suitable for serverless deployments)
Use command line
sdap-cmd --help
For example with EVI
operator:
sdap-cmd --conf ./src/sdap/test/data_access/collection-config.yaml \
--collection hls \
--x-range -71.24 -71.23 \
--y-range 42.40 42.42 \
--time-range 2017-01-01T00:00:00.000000+00:00 2017-06-01T00:00:00.000000+00:00 \
--operator-name EVI \
--operator-args '[[0,0,-2.5,2.5,0,0,0],[0,0,2.4,1,0,0,1]]' \
--plot
With SpatialMean
operator:
sdap-cmd --conf ./src/sdap/test/data_access/collection-config.yaml \
--secrets my_secrets.yaml \
--collection hls \
--bbox -71.24 42.40 -71.23 42.42 \
--time-range 2017-01-01T00:00:00.000000+00:00 2017-06-01T00:00:00.000000+00:00 \
--operator-name SpatialMean
sdap-cmd --conf ./src/sdap/test/data_access/collection-config.yaml \
--collection maiac \
--bbox -71.25 42.39 -71.23 42.42 \
--time-range 2019-01-01T00:00:00.000000+00:00 2019-02-01T00:00:00.000000+00:00 \
--operator-name SpatialMean
As a service
Starts a web server with an API:
sdap-serv --conf src/sdap/test/data_access/collection-config.yaml --secrets my_secrets.yaml
Test the service, you can use the jupyter notebook(s) provided on src/sdap/notebook
You can run it locally by running:
pip install -e '.[client]'
jupyter-lab
Then select the notebook in the jupyuter web console and execute it, step by step.
Resource allocation
The resources needed to run the application can be estimated as follows:
- maximum size of 1 tile for each worker = T = 64 * x * y * t * obs dim, for example 643660366016 = 5Gb
- maximum size of the user result returned to the user = R = 64xytobs dims
- number of parallels workers: n
Total RAM = T*n + R
Number of CPU = n
For developers
pip install -e '.[dev]'
Do you work, aad unit tests
Validate
python setup.py test
Package
Upgrade version in `src/sdap/VERSION.txt`
python setup.py sdist bdist_wheel
Create and publish the docker image
docker build . -t tloubrieu/sdap-os:0.1.1
docker push tloubrieu/sdap-os:0.1.1
Create the kubernetes namespace:
kubectl create namespace sdap-os
Deploy the ray.io operator:
(See instructions on https://docs.ray.io/en/latest/cluster/kubernetes.html#ray-helm)
git clone https://github.com/ray-project/ray.git
cd ray/deploy/charts
helm -n ray install sdap-cluster --create-namespace ./ray
Deploy the helm chart
helm install sdap-os ./helm/sdap-os --dependency-update -n sdap-os
Stop the service:
helm delete sdap-os -n sdap-os
On a local deployment with single node, the node might be tainted to prevent pods from being deployed on it. In this case you can run something like:
kubectl taint node docker-desktop node.kubernetes.io/disk-pressure:NoSchedule-
(see https://stackoverflow.com/questions/62991596/1-nodes-had-taints-that-the-pod-didnt-tolerate-in-kubernetes-cluster for more details)
Use with local minio
Set minio server (not used in the development yet)
Standalone minio:
Server install:
brew install minio/stable/minio
Start the server
export MINIO_ROOT_USER=myminio
export MINIO_ROOT_PASSWORD=minio123AB
minio server /Users/loubrieu/Documents/sdap/minio --address localhost:9000 --console-address localhost:19000
Client
brew install minio/stable/mc
mc --help
Create server alias:
mc alias set minio http://localhost:9000 myminio minio123AB
Create a bucket:
mc mb hls
Copy test datasets
Create a bucket:
mc mb hls
Copy some files:
mc cp data/hls/s1_output_latlon_HLS_S30_T18TYN_2019263_cog.tif minio/hls
mc cp data/hls/s1_output_latlon_HLS_S30_T18TYN_2019260_cog.tif minio/hls
mc cp data/hls/s1_output_latlon_HLS_S30_T18TYN_2019253_cog.tif minio/hls
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.