The ICEBERG Rivers usecase package
Project description
Rivers (Arctic hydrology)
We provide a classification algorithm for ice surface features from high-resolution imagery. This algorithm was developed by convolutional neural network training to detect regions of large and small rivers and to distinguish them from crevasses and non-channelized slush. We also provide a detection algorithm to extract polyline water features from the classified high-probability river areas.
Prerequisites - all available on bridges via the commands below
- Linux
- Python 3
- CPU and NVIDIA GPU + CUDA CuDNN
Software Dependencies - these will be installed automatically with the installation below.
- numpy
- scipy
- tifffile
- keras==2.2.4
- opencv-python
- rasterio
- affine
- pygdal==2.2.1
- tensorflow-gpu==1.7.1
Installation
Preliminaries:
These instructions are specific to XSEDE Bridges but other resources can be used if cuda, python3, and a NVIDIA P100 GPU are present, in which case 'module load' instructions can be skipped, which are specific to Bridges.
For Unix or Mac Users:
Login to bridges via ssh using a Unix or Mac command line terminal. Login is available to bridges directly or through the XSEDE portal. Please see the Bridges User's Guide.
For Windows Users:
Many tools are available for ssh access to bridges. Please see Ubuntu, MobaXterm or PuTTY
PSC Bridges 2
Once you have logged into Bridges 2, you can follow one of two methods for installing iceberg-rivers.
Method 1 (Recommended):
The lines below following a '$' are commands to enter (or cut and paste) into your terminal (note that all commands are case-sensitive, meaning capital and lowercase letters are differentiated.) Everything following '#' are comments to explain the reason for the command and should not be included in what you enter. Lines that do not start with '$' or '[rivers_env] $' are output you should expect to see.
$ pwd
/home/username
$ cd $PROJECT # switch to your working space.
$ mkdir Rivers # create a directory to work in.
$ cd Rivers # move into your working directory.
$ module load cuda/9.2.0
$ virtualenv rivers_env --system-site-packages # create a virtual environment to isolate your work from the default system.
$ source rivers_env/bin/activate # activate your environment. Notice the command line prompt changes to show your environment on the next line.
[rivers_env] $ pwd
/ocean/projects/group/username/Rivers
[rivers_env] $ export PYTHONPATH=<path>/rivers_env/lib/python3.6/site-packages:$PYTHONPATH # set a system variable to point python to your specific code. (Replace <path> with the results of pwd command above.
[rivers_env] $ pip install iceberg_rivers.search # pip is a python tool to extract the requested software (iceberg_rivers.search in this case) from a repository. (this may take several minutes).
Method 2 (Installing from source; recommended for developers only):
$ git clone https://github.com/iceberg-project/Rivers.git
$ module load cuda/9.2.0
$ virtualenv rivers_env --system-site-packages
$ source rivers_env/bin/activate
[rivers_env] $ export PYTHONPATH=<path>/rivers_env/lib/python3.6/site-packages:$PYTHONPATH
[rivers_env] $ pip install . --upgrade
To test
[iceberg_rivers] $ deactivate # exit your virtual environment.
$ interact -p GPU-small --gres=gpu:p100:1 # request a compute node. This package has been tested on P100 GPUs on bridges, but that does not exclude any other resource that offers the same GPUs. (this may take a minute or two or more to receive an allocation).
$ cd $PROJECT/Rivers # make sure you are in the same directory where everything was set up before.
$ module load keras/2.2.0_tf1.7_py3_gpu gdal/2.2.1 # load keras libraries and GDAL, as before.
$ source rivers_env/bin/activate # activate your environment, no need to create a new environment because the Rivers tools are installed and isolated here.
[rivers_env] $ export PYTHONPATH=<path>/rivers_env/lib/python3.6/site-packages:$PYTHONPATH
[rivers_env] $ iceberg_rivers.tiling --help # this will display a help screen of available usage and parameters.
Prediction
- Download a pre-trained model at:
You can download to your local machine and use scp, ftp, rsync, or Globus to transfer to bridges.
Rivers predicting is executed in three steps: First, follow the environment setup commands under 'To test' above. Then create tiles from an input GeoTiff image and write to the output_folder. The scale_bands parameter (in pixels) depends on the trained model being used. The default scale_bands is 299 for the pre-trained model downloaded above. If you use your own model the scale_bands may be different.
[rivers_env] $ iceberg_rivers.tiling --tile_size=224 --step=112 --input=<image_abspath> --output=./test/
Then, detect rivers on each tile and output counts and confidence for each tile.
[rivers_env] $ iceberg_rivers.predict --input <tile_folder> -o <output_folder> -w <model>
Finally, mosaic all the tiles back into one image
[rivers_env] $ iceberg_rivers.mosaic --input_WV image --input <masks_folder> --tile_size 224 --step 112 --output_folder ./mosaic
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Hashes for iceberg_rivers.search-0.1.3.dev3.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8be2ad77cb0b7e88afb7a300ff1ecfe10fe4dc8003b936baf1dd44df0468dde3 |
|
MD5 | a90827b380616231c640e4fde0ad3bfb |
|
BLAKE2b-256 | 5d091e11d47a0fd86d7233e10d29bfc0053bf56f7d193adfe088821fa22abdd6 |