Skip to content

Commit b6e6d79

Browse files
kasra-keshavarzkasra-keshavarz
authored
gistool v0.1.0-rc1 release (#1)
* creating dev branch * parsing MERIT-Hydro extents * modifications on MERIT-Hydro script and validating its output * adding .gitignore and details * adding stats extraction feature without testing * adding renv setup and files * debuggin and adding stats and quantiles capabilities * finalizing merit-hydro * finalizing merit-hydro stats and subsetting * Changing GDAL versin from 3.0.4 to 3.4.1 The change is due to complete compatibility with all the datasets included in this tool, mainly the MODIS MCDQ12A (land cover) dataset that works best with the 3.0.4 version. * Adding the capability to subset and implement efficient zonal statistics on SoilGridsV1 GeoTIFFs The file reads the .tif files, geographically subsets the files based on the given latitude and longitude extents, and prints given input `--stats` in a .csv file. The users are expected to enter complete variable names taken from the GitHub repository README page for this specific datasets. Reported by: Kasra Keshavarz Signed-off-by: Kasra Keshavarz <kasra.keshavarz@usask.ca> * Debugging problems with zonal statistics and adding log date for errors The zonal statistics were migrated from merit-hydro scripts without modifications, so necessary editions were implemented to conform to the SoilGridsV1 dataset and its nomenclature. The log date has been prepended to the program-generated errors and warnings which is useful for future debuggings, etc. Reported by: Kasra Keshavarz Signed-off-by: Kasra Keshavarz <kasra.keshavarz@usask.ca> * initial README.md file for SoilGrids dataset * README file created for MODIS dataset * missing logDate from previous commit added now * Adding MODIS zonal statistics and GeoTIFF Subsetting feature MODIS landcover provides valuable information for setting up hydrological models and therefore it has been added to the repository. The zonal statistics using `exactextractr` is very efficient and produces `frac` of each land cover class. Apart from that, all other statistics that are available to other datasets, could be used with this dataset as well. Reported by: Kasra Keshavarz Signed-off-by: Kasra Keshavarz <kasra.keshavarz@usask.ca> * removing debugging lines * adding initial information for the main README page * renaming for better clarity of the tool * adding few options to take into account shapefiles that does not have a CRS defined * renaming * initial example and README files * initial README file * merit-hydro example initialized * adding the job submission option to the example * initial soil grids example added * modis example initialized * typos corrected * fixing typos * Added description of README file for future reference * adding license header * relevant info added to the README file * typos and corrections * adding no verbose option to wget download * adding missing backslashes for line continuation * adding quiet option to wget * typos * correcting wget options orders * fixing job submission cache path typos * adding full path to the shapefile argument * necessary technical details of the MERIT-Hydro dataset * typos, correction, and adding file contents of one .tar file as an example * Added necessary technical information to the READMEs * necessary extra info added * correcting soil_grids directory address typo * fixing typos
1 parent 861f932 commit b6e6d79

File tree

17 files changed

+2102
-328
lines changed

17 files changed

+2102
-328
lines changed

‎.gitignore

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
*.pyc
2+
.git
3+
.ipynb_checkpoints
4+
.DS_Store
5+
*.swp

‎README.md

Lines changed: 43 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -1,68 +1,79 @@
11
# Description
2-
This repository contains scripts to process necessary GeoTIFF datasets. The general usage of the script (i.e., `./extract-geotiff.sh`) is as follows:
2+
This repository contains scripts to process necessary geospatial datasets and implement efficient zonal statistics on given ESRI Shapefiles. The general usage of the script (i.e., `./extract-gis.sh`) is as follows:
33

44
```console
55
Usage:
6-
extract-geotiff [options...]
6+
extract-gis [options...]
77

88
Script options:
9-
-d, --dataset GeoTIFF dataset of interest,
10-
currently available options are:
11-
'MODIS';'MERIT-Hydro';'SoilGridsV1';
12-
'SoilGridsV2';
9+
-d, --dataset Geospatial dataset of interest, currently
10+
available options are: 'MODIS';
11+
'MERIT-Hydro';'SoilGridsV1'
1312
-i, --dataset-dir=DIR The source path of the dataset file(s)
13+
-r, --crs=INT The EPSG code of interest; optional
14+
[defaults to 4326]
1415
-v, --variable=var1[,var2[...]] If applicable, variables to process
1516
-o, --output-dir=DIR Writes processed files to DIR
16-
-s, --start-date=DATE If applicable, start date of the GeoTIFF
17+
-s, --start-date=DATE If applicable, start date of the geospatial
1718
data; optional
18-
-e, --end-date=DATE If applicable, end date of the GeoTIFF
19+
-e, --end-date=DATE If applicable, end date of the geospatial
1920
data; optional
2021
-l, --lat-lims=REAL,REAL Latitude's upper and lower bounds; optional
2122
-n, --lon-lims=REAL,REAL Longitude's upper and lower bounds; optional
22-
-p, --shape-file=PATH Path to the ESRI '.shp' file; optional
23+
-f, --shape-file=PATH Path to the ESRI '.shp' file; optional
2324
-j, --submit-job Submit the data extraction process as a job
2425
on the SLURM system; optional
25-
-t, --stats=stat1[,stat2[...]] If applicable, extract the statistics of
26+
-t, --print-geotiff=BOOL Extract the subsetted GeoTIFF file; optional
27+
[defaults to 'true']
28+
-a, --stat=stat1[,stat2[...]] If applicable, extract the statistics of
2629
interest, currently available options are:
2730
'min';'max';'mean';'majority';'minority';
28-
'median';'quantiles';'variety';'variance';
31+
'median';'quantile';'variety';'variance';
2932
'stdev';'coefficient_of_variation';'frac';
33+
optional
34+
-q, --quantile=q1[,q2[...]] Quantiles of interest to be produced if 'quantile'
35+
is included in the '--stat' argument. The values
36+
must be comma delimited float numbers between
37+
0 and 1; optional [defaults to every 5th quantile]
3038
-p, --prefix=STR Prefix prepended to the output files
3139
-c, --cache=DIR Path of the cache directory; optional
32-
-E, --email=STR E-mail when job starts, ends, and
40+
-E, --email=STR E-mail when job starts, ends, and
3341
finishes; optional
3442
-V, --version Show version
3543
-h, --help Show this screen and exit
3644
```
3745

3846

3947
# Available Datasets
40-
|**#**|Dataset |Time Scale |DOI |Description |
41-
|-----|--------------------------------------------|----------------------|-----------------------|---------------------|
42-
|**1**|MODIS |2000 - 2021 | |[link](modis) |
43-
|**2**|MERIT Hydro |Not Applicable (N/A) |10.1029/2019WR024873 |[link](merit_hydro) |
44-
|**3**|Soil Grids (v1) | N/A |*ditto* |[link](soil_grids_v1)|
45-
|**4** |Soil Grids (v2) | N/A |*ditto* |[link](soil_grids_v2)|
48+
|**#**|Dataset |Time Scale |CRS |DOI |Description |
49+
|-----|--------------------------------------------|----------------------|-----|-------------------------------|---------------------|
50+
|**1**|MODIS |2000 - 2021 | |10.5067/MODIS/MCD12Q1.006 |[link](modis) |
51+
|**2**|MERIT Hydro |Not Applicable (N/A) |4326 |10.1029/2019WR024873 |[link](merit_hydro) |
52+
|**3**|Soil Grids (v1) |Not Applicable (N/A) |4326 |10.1371/journal.pone.0169748 |[link](soil_grids)|
4653

4754

4855
# General Example
4956
As an example, follow the code block below. Please remember that you MUST have access to Graham cluster with Compute Canada (CC) and have access to `MERIT-Hydro` dataset. Also, remember to generate a [Personal Access Token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) with GitHub in advance. Enter the following codes in your Graham shell as a test case:
5057

5158
```console
52-
foo@bar:~$ git clone https://github.com/kasra-keshavarz/geotifftool # clone the repository
53-
foo@bar:~$ cd ./geotifftool/ # always move to the repository's directory
54-
foo@bar:~$ ./extract-geotiff.sh -h # view the usage message
55-
foo@bar:~$ ./extract-geotiff.sh --geotiff="MERIT-Hydro" \
56-
--dataset-dir="/project/rpp-kshook/Model_Output/WRF/CONUS/CTRL" \
57-
--output-dir="$HOME/scratch/conus_i_output/" \
58-
--start-date="2001-01-01 00:00:00" \
59-
--end-date="2001-12-31 23:00:00" \
60-
--lat-lims=49,51 \
61-
--lon-lims=-117,-115 \
62-
--variable=T2,PREC_ACC_NC,Q2,ACSWDNB,ACLWDNB,U10,V10,PSFC \
63-
--prefix="conus_i";
59+
foo@bar:~$ git clone https://github.com/kasra-keshavarz/gistool # clone the repository
60+
foo@bar:~$ cd ./gistool/ # always move to the repository's directory
61+
foo@bar:~$ ./extract-gis.sh -h # view the usage message
62+
foo@bar:~$ wget -m -nd -nv -q -A "cat_pfaf_71_MERIT_Hydro_v07_Basins_v01_bugfix1.*" \
63+
"http://hydrology.princeton.edu/data/mpan/MERIT_Basins/MERIT_Hydro_v07_Basins_v01_bugfix1/pfaf_level_02/";
64+
# downloading a sample shapefile
65+
foo@bar:~$ ./extract-gis.sh --dataset="merit-hydro" \
66+
--dataset-dir="/project/rpp-kshook/CompHydCore/merit_hydro/raw_data/" \
67+
--output-dir="$HOME/scratch/merit-hydro-test" \
68+
--shape-file="./cat_pfaf_67_MERIT_Hydro_v07_Basins_v01_bugfix1.shp" \
69+
--print-geotiff=true \
70+
--stat="min,max,mean,median,quantile" \
71+
--quantile="0.1,0.5,0.9" \
72+
--variable="elv,hnd" \
73+
--prefix="merit_test_";
74+
6475
```
65-
See the [example](./example) directory for real-world scripts for each GeoTIFF dataset included in this repository.
76+
See the [example](./example) directory for real-world scripts for each geospatial dataset included in this repository.
6677

6778

6879
# New Datasets
@@ -74,7 +85,7 @@ Please open a new ticket on the **Issues** tab of the current repository in case
7485

7586

7687
# License
77-
GeoTIFF Processing Workflow<br>
88+
Geospatial Dataset Processing Workflow<br>
7889
Copyright (C) 2022, University of Saskatchewan<br>
7990

8091
This program is free software: you can redistribute it and/or modify

‎assets/README.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# Description
2+
3+
This directory contains two main files: 1) `renv.lock` containing necessary meta-data for the R `renv` package to set up the environment/libraries necessary for running the [`exactextractr`](https://github.com/isciences/exactextractr) package that implements zonal statistics and 2) `stats.R` that calls [`exactextractr`](https://github.com/isciences/exactextractr) package after the R environment has been fully set up.
4+

0 commit comments

Comments
 (0)