Skip to content

Commit

Permalink
Squashed commit of the following:
Browse files Browse the repository at this point in the history
commit a92212d
Author: TrevorGrout-NOAA <69653333+TrevorGrout-NOAA@users.noreply.github.com>
Date:   Thu Feb 25 12:45:26 2021 -0600

    Enhancement that creates metric plots and summary statistics.

    Enhancement that creates metric plots and summary statistics using metrics compiled by synthesize_test_cases.py.

    Additions
    - Added eval_plots.py, which produces:
    - Boxplots of CSI, FAR, and POD/TPR
    - Barplot of aggregated CSI scores
    - Scatterplot of CSI comparing two FIM versions
    - CSV of aggregated statistics (CSI, FAR, POD/TPR)
    - CSV of analyzed data and analyzed sites

    This resolves #70.

commit ca68f2b
Author: Brad <bradford.bates@noaa.gov>
Date:   Thu Feb 25 08:37:21 2021 -0600

    Update CHANGELOG.md

commit e2ae250
Author: Brad <bradford.bates@noaa.gov>
Date:   Wed Feb 24 10:58:31 2021 -0600

    Bug fixes to new evaluation code and README.md cleanup

    - Fixed a bug in synthesize_test_cases.py where the extent (MS/FR) was not being written to merged metrics file properly.
    - Fixed a bug in synthesize_test_cases.py where only BLE test cases were being written to merged metrics file.
    - Removed unused imports from inundation.py.
    - Updated README.md

    This resolves #270.

commit ffa0a00
Author: RyanSpies-NOAA <ryan.spies@noaa.gov>
Date:   Tue Feb 23 13:59:17 2021 -0600

    Adding HAND SRC datum elev values to hydroTable.csv output (#272)

    - Adding HAND SRC datum elev values to hydroTable.csv output
    - Renamed hydroid attribute to "Median_Thal_Elev_m"

commit 1898e93
Author: Brad <bradford.bates@noaa.gov>
Date:   Tue Feb 23 09:05:15 2021 -0600

    Update README.md

commit 31af057
Author: Brad <bradford.bates@noaa.gov>
Date:   Tue Feb 23 09:04:19 2021 -0600

    Update README.md

commit 48e2359
Author: Brad <bradford.bates@noaa.gov>
Date:   Tue Feb 23 09:03:31 2021 -0600

    Update README.md

commit 9782453
Author: Brad <bradford.bates@noaa.gov>
Date:   Tue Feb 23 09:02:23 2021 -0600

    Update README.md

commit e0276ad
Author: Brad <bradford.bates@noaa.gov>
Date:   Tue Feb 23 09:00:23 2021 -0600

    Update README.md

commit a683d8b
Author: Brad <bradford.bates@noaa.gov>
Date:   Tue Feb 23 08:59:04 2021 -0600

    Update README.md

commit ad97abb
Author: Brad <bradford.bates@noaa.gov>
Date:   Tue Feb 23 08:50:21 2021 -0600

    Update README.md

commit 4d42ccc
Author: Brad <bradford.bates@noaa.gov>
Date:   Tue Feb 23 08:26:01 2021 -0600

    Update README.md

commit 1640e26
Author: Brad <bradford.bates@noaa.gov>
Date:   Tue Feb 23 08:23:50 2021 -0600

    Update README.md

commit 6c1e6a3
Author: Brad <bradford.bates@noaa.gov>
Date:   Tue Feb 23 08:21:43 2021 -0600

    Update README.md

commit 747deaf
Author: Brad <bradford.bates@noaa.gov>
Date:   Tue Feb 23 08:19:36 2021 -0600

    Update README.md

commit 83f9a3b
Author: Brad <bradford.bates@noaa.gov>
Date:   Tue Feb 23 08:12:46 2021 -0600

    Update README.md

commit 90dc695
Author: Brad <bradford.bates@noaa.gov>
Date:   Tue Feb 23 08:11:44 2021 -0600

    Update README.md

commit 0347e63
Author: Brad <bradford.bates@noaa.gov>
Date:   Tue Feb 23 07:52:17 2021 -0600

    Update README.md

commit 21d05c6
Author: Brad <bradford.bates@noaa.gov>
Date:   Mon Feb 22 18:52:45 2021 -0600

    Update README.md

commit 6d17e9a
Author: Brad <bradford.bates@noaa.gov>
Date:   Mon Feb 22 18:33:34 2021 -0600

    Update README.md

commit 08a4b98
Author: Brad <bradford.bates@noaa.gov>
Date:   Mon Feb 22 17:09:36 2021 -0600

    Update README.md

commit 6df54a5
Author: Brad <bradford.bates@noaa.gov>
Date:   Mon Feb 22 17:08:00 2021 -0600

    Update README.md

commit 8e2ccc2
Author: Brad <bradford.bates@noaa.gov>
Date:   Mon Feb 22 10:29:23 2021 -0600

    Update CHANGELOG.md

commit 7db1284
Author: Brad <bradford.bates@noaa.gov>
Date:   Mon Feb 22 09:27:01 2021 -0600

    Removed "_new" from test_cases dir

commit 584f4a3
Author: Brad <bradford.bates@noaa.gov>
Date:   Mon Feb 22 09:22:43 2021 -0600

    Enhancements to allow for evaluation at AHPS sites, the generation of a query-optimized metrics CSV, and the generation of categorical FIM.

    Enhancements to allow for evaluation at AHPS sites, the generation of a query-optimized metrics CSV, and the generation of categorical FIM. This merge requires that the `/test_cases` directory be updated for all machines performing evaluation.

     - `generate_categorical_fim.py` was added to allow production of NWS Flood Categorical HAND FIM (CatFIM) source data. More changes on this script are to follow in subsequent branches.
     - `ble_autoeval.sh` and `all_ble_stats_comparison.py` were deleted because `synthesize_test_cases.py` now handles the merging of metrics.
     - The code block in `run_test_case.py` that was responsible for printing the colored metrics to screen has been commented out because of the new scale of evaluations (formerly in `run_test_case.py`, now in `shared_functions.py`)
     - Remove unused imports from inundation wrappers in `/tools`.
     - Updated `synthesize_test_cases.py` to allow for AHPS site evaluations.
     - Reorganized `run_test_case.py` by moving more functions into `shared_functions.py`.
     - Created more shared variables in `shared_variables.py` and updated import statements in relevant scripts.

    This resolves #258 and #259.

commit 0412e41
Author: brian.avant <brian.avant@noaa.gov>
Date:   Fri Feb 19 12:29:17 2021 -0600

    Rating curves for short stream segments are replaced with rating curves from upstream/downstream segments.

     - Short stream segments are identified and are reassigned the channel geometry from upstream/downstream segment.
     - fossid renamed to fimid and the attribute's starting value is now 1000 to avoid HydroIDs with leading zeroes.
     - Addresses issue where HydroIDs were not included in final hydrotable.
     - Added import sys to inundation.py (missing from previous feature branch).
     - Variable names and general workflow are cleaned up.

    This resolves #100.

commit d615a6c
Author: RyanSpies-NOAA <ryan.spies@noaa.gov>
Date:   Fri Feb 12 15:59:00 2021 -0600

     Modified rem.py with a new function to output HAND reference elev

    - Function "make_catchment_hydroid_dict" creates a df of pixel catchment ids and overlapping hydroids
    - Merge hydroid df and thalweg minimum elevation df
    - Produces new output containing all catchment ids and min thalweg elevation value → “hand_ref_elev_table.csv”
    - Overwrites the demDerived_reaches_split.gpk layer --> adding additional attribute: “Min_Thal_Elev_meters” to view the elevation value for each hydroid

commit f474736
Author: brian.avant <brian.avant@noaa.gov>
Date:   Fri Feb 12 15:35:11 2021 -0600

    Addresses issue when running on HUC6 scale

    * src should be fixed and slightly smaller by removing whitespace
    * rasters are about the same size as running fim as huc6 (compressed and tiled; aggregated are slightly larger)
    * naming convention and feature id attribute are only added to the aggregated hucs (this bullet and the next one assume that Viz will not use the huc8 layer outputs)
    * HydroIDs are different for huc6 vs aggregated huc8s mostly due to forced split at huc boundaries (so long we use consistent workflow it shouldn't matter)
    * known issue where sometimes an incoming stream is not included in the final selection (resolves #238) will affect aggregate outputs

commit ad89e6d
Author: brian.avant <brian.avant@noaa.gov>
Date:   Fri Feb 12 08:57:03 2021 -0600

    Updated MS Crosswalk method to address gaps in FIM.

    - Fixed typo in stream midpoint calculation in split_flows.py and add_crosswalk.py.
    - add_crosswalk.py now restricts the MS crosswalk to NWM MS catchments.
    - add_crosswalk.py now performs a secondary MS crosswalk selection by nearest NWM MS catchment.

    This resolves #232.

commit 57cb283
Author: Brad <bradford.bates@noaa.gov>
Date:   Wed Feb 10 14:01:24 2021 -0600

    Update README.md

commit b147f45
Author: Brad <bradford.bates@noaa.gov>
Date:   Wed Feb 10 14:00:16 2021 -0600

    Update README.md

commit 2ffd97b
Author: RyanSpies-NOAA <ryan.spies@noaa.gov>
Date:   Wed Feb 10 09:46:29 2021 -0600

    New python script "wrappers" for using inundation.py.

    - Created inundation_wrapper_nwm_flows.py to produce inundation outputs using NWM recurrence flows: 1.5 year, 5 year, 10 year.
    - Created inundation_wrapper_custom_flow.py to produce inundation outputs with user-created flow file.
    - Created new tools parent directory to store inundation_wrapper_nwm_flows.py and inundation_wrapper_custom_flow.py.

    This resolves #220.
  • Loading branch information
fernando-aristizabal committed Mar 1, 2021
1 parent 25e556d commit 4b47431
Show file tree
Hide file tree
Showing 30 changed files with 2,538 additions and 809 deletions.
134 changes: 134 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,132 @@
All notable changes to this project will be documented in this file.
We follow the [Semantic Versioning 2.0.0](http://semver.org/) format.
<br/><br/>
## v3.0.6.0 - 2021-02-25 - [PR #276](https://github.com/NOAA-OWP/cahaba/pull/276)

Enhancement that creates metric plots and summary statistics using metrics compiled by `synthesize_test_cases.py`.

### Additions
- Added `eval_plots.py`, which produces:
- Boxplots of CSI, FAR, and POD/TPR
- Barplot of aggregated CSI scores
- Scatterplot of CSI comparing two FIM versions
- CSV of aggregated statistics (CSI, FAR, POD/TPR)
- CSV of analyzed data and analyzed sites

<br/><br/>
## v3.0.5.3 - 2021-02-23 - [PR #275](https://github.com/NOAA-OWP/cahaba/pull/275)

Bug fixes to new evaluation code.

### Changes

- Fixed a bug in `synthesize_test_cases.py` where the extent (MS/FR) was not being written to merged metrics file properly.
- Fixed a bug in `synthesize_test_cases.py` where only BLE test cases were being written to merged metrics file.
- Removed unused imports from `inundation.py`.
- Updated README.md

<br/><br/>
## v3.0.5.2 - 2021-02-23 - [PR #272](https://github.com/NOAA-OWP/cahaba/pull/272)

Adds HAND synthetic rating curve (SRC) datum elevation values to `hydroTable.csv` output.

### Changes

- Updated `add_crosswalk.py` to included "Median_Thal_Elev_m" variable outputs in `hydroTable.csv`.
- Renamed hydroid attribute in `rem.py` to "Median" in case we want to include other statistics in the future (e.g. min, max, range etc.).

<br/><br/>
## v3.0.5.1 - 2021-02-22

Fixed `TEST_CASES_DIR` path in `tests/utils/shared_variables.py`.

### Changes

- Removed `"_new"` from `TEST_CASES_DIR` variable.

<br/><br/>
## v3.0.5.0 - 2021-02-22 - [PR #267](https://github.com/NOAA-OWP/cahaba/pull/267)

Enhancements to allow for evaluation at AHPS sites, the generation of a query-optimized metrics CSV, and the generation of categorical FIM. This merge requires that the `/test_cases` directory be updated for all machines performing evaluation.

### Additions

- `generate_categorical_fim.py` was added to allow production of NWS Flood Categorical HAND FIM (CatFIM) source data. More changes on this script are to follow in subsequent branches.

### Removals

- `ble_autoeval.sh` and `all_ble_stats_comparison.py` were deleted because `synthesize_test_cases.py` now handles the merging of metrics.
- The code block in `run_test_case.py` that was responsible for printing the colored metrics to screen has been commented out because of the new scale of evaluations (formerly in `run_test_case.py`, now in `shared_functions.py`)
- Remove unused imports from inundation wrappers in `/tools`.

### Changes

- Updated `synthesize_test_cases.py` to allow for AHPS site evaluations.
- Reorganized `run_test_case.py` by moving more functions into `shared_functions.py`.
- Created more shared variables in `shared_variables.py` and updated import statements in relevant scripts.

<br/><br/>

## v3.0.4.4 - 2021-02-19 - [PR #266](https://github.com/NOAA-OWP/cahaba/pull/266)

Rating curves for short stream segments are replaced with rating curves from upstream/downstream segments.

### Changes

- Short stream segments are identified and are reassigned the channel geometry from upstream/downstream segment.
- `fossid` renamed to `fimid` and the attribute's starting value is now 1000 to avoid HydroIDs with leading zeroes.
- Addresses issue where HydroIDs were not included in final hydrotable.
- Added `import sys` to `inundation.py` (missing from previous feature branch).
- Variable names and general workflow are cleaned up.

<br/><br/>
## v3.0.4.3 - 2021-02-12 - [PR #254](https://github.com/NOAA-OWP/cahaba/pull/254)

Modified `rem.py` with a new function to output HAND reference elev.

### Changes

- Function `make_catchment_hydroid_dict` creates a df of pixel catchment ids and overlapping hydroids.
- Merge hydroid df and thalweg minimum elevation df.
- Produces new output containing all catchment ids and min thalweg elevation value named `hand_ref_elev_table.csv`.
- Overwrites the `demDerived_reaches_split.gpk` layer by adding additional attribute `Min_Thal_Elev_meters` to view the elevation value for each hydroid.

<br/><br/>
## v3.0.4.2 - 2021-02-12 - [PR #255](https://github.com/NOAA-OWP/cahaba/pull/255)

Addresses issue when running on HUC6 scale.

### Changes

- `src.json` should be fixed and slightly smaller by removing whitespace.
- Rasters are about the same size as running fim as huc6 (compressed and tiled; aggregated are slightly larger).
- Naming convention and feature id attribute are only added to the aggregated hucs.
- HydroIDs are different for huc6 vs aggregated huc8s mostly due to forced split at huc boundaries (so long we use consistent workflow it shouldn't matter).
- Fixed known issue where sometimes an incoming stream is not included in the final selection will affect aggregate outputs.

<br/><br/>
## v3.0.4.1 - 2021-02-12 - [PR #261](https://github.com/NOAA-OWP/cahaba/pull/261)

Updated MS Crosswalk method to address gaps in FIM.

### Changes

- Fixed typo in stream midpoint calculation in `split_flows.py` and `add_crosswalk.py`.
- `add_crosswalk.py` now restricts the MS crosswalk to NWM MS catchments.
- `add_crosswalk.py` now performs a secondary MS crosswalk selection by nearest NWM MS catchment.

<br/><br/>
## v3.0.4.0 - 2021-02-10 - [PR #256](https://github.com/NOAA-OWP/cahaba/pull/256)

New python script "wrappers" for using `inundation.py`.

### Additions

- Created `inundation_wrapper_nwm_flows.py` to produce inundation outputs using NWM recurrence flows: 1.5 year, 5 year, 10 year.
- Created `inundation_wrapper_custom_flow.py` to produce inundation outputs with user-created flow file.
- Created new `tools` parent directory to store `inundation_wrapper_nwm_flows.py` and `inundation_wrapper_custom_flow.py`.

<br/><br/>
## v3.0.3.1 - 2021-02-04 - [PR #253](https://github.com/NOAA-OWP/cahaba/pull/253)

Bug fixes to correct mismatched variable name and file path.
Expand All @@ -10,6 +136,7 @@ Bug fixes to correct mismatched variable name and file path.
- Corrected variable name in `fim_run.sh`.
- `acquire_and_preprocess_inputs.py` now creates `huc_lists` folder and updates file path.

<br/><br/>
## v3.0.3.0 - 2021-02-04 - [PR #227](https://github.com/NOAA-OWP/cahaba/pull/227)

Post-process to aggregate FIM outputs to HUC6 scale.
Expand All @@ -28,6 +155,7 @@ Post-process to aggregate FIM outputs to HUC6 scale.
- `build_stream_traversal.py` is now assigning HydroID by midpoint instead centroid.
- Cleanup of `clip_vectors_to_wbd.py`.

<br/><br/>
## v3.0.2.0 - 2021-01-25 - [PR #218](https://github.com/NOAA-OWP/cahaba/pull/218)

Addition of an API service to schedule, run and manage `fim_run` jobs through a user-friendly web interface.
Expand All @@ -36,6 +164,7 @@ Addition of an API service to schedule, run and manage `fim_run` jobs through a

- `api` folder that contains all the codebase for the new service.

<br/><br/>
## v3.0.1.0 - 2021-01-21 - [PR #206](https://github.com/NOAA-OWP/cahaba/pull/206)

Preprocess MS and FR stream networks
Expand All @@ -51,6 +180,7 @@ Preprocess MS and FR stream networks
- Added `getDriver` to shared `functions.py`.
- Cleaned up variable names and types.

<br/><br/>
## v3.0.0.4 - 2021-01-20 - [PR #230](https://github.com/NOAA-OWP/cahaba/pull/230)

Changed the directory where the `included_huc*.lst` files are being read from.
Expand All @@ -59,6 +189,7 @@ Changed the directory where the `included_huc*.lst` files are being read from.

- Changed the directory where the `included_huc*.lst` files are being read from.

<br/><br/>
## v3.0.0.3 - 2021-01-14 - [PR #210](https://github.com/NOAA-OWP/cahaba/pull/210)

Hotfix for handling nodata value in rasterized levee lines.
Expand All @@ -68,6 +199,7 @@ Hotfix for handling nodata value in rasterized levee lines.
- Resolves bug for HUCs where `$ndv > 0` (Great Lakes region).
- Initialize the `nld_rasterized_elev.tif` using a value of `-9999` instead of `$ndv`.

<br/><br/>
## v3.0.0.2 - 2021-01-06 - [PR #200](https://github.com/NOAA-OWP/cahaba/pull/200)

Patch to address AHPSs mapping errors.
Expand All @@ -79,6 +211,7 @@ Patch to address AHPSs mapping errors.
- Updates path to latest AHPs site layer.
- Updated [readme](https://github.com/NOAA-OWP/cahaba/commit/9bffb885f32dfcd95978c7ccd2639f9df56ff829)

<br/><br/>
## v3.0.0.1 - 2020-12-31 - [PR #184](https://github.com/NOAA-OWP/cahaba/pull/184)

Modifications to build and run Docker image more reliably. Cleanup on some pre-processing scripts.
Expand All @@ -91,6 +224,7 @@ Modifications to build and run Docker image more reliably. Cleanup on some pre-p
### Notes
- `aggregate_vector_inputs.py` doesn't work yet. Need to externally download required data to run fim_run.sh

<br/><br/>
## v3.0.0.0 - 2020-12-22 - [PR #181](https://github.com/NOAA-OWP/cahaba/pull/181)

The software released here builds on the flood inundation mapping capabilities demonstrated as part of the National Flood Interoperability Experiment, the Office of Water Prediction's Innovators Program and the National Water Center Summer Institute. The flood inundation mapping software implements the Height Above Nearest Drainage (HAND) algorithm and incorporates community feedback and lessons learned over several years. The software has been designed to meet the requirements set by stakeholders interested in flood prediction and has been developed in partnership with several entities across the water enterprise.
Loading

0 comments on commit 4b47431

Please sign in to comment.