Merge branch 'dev' into dev-external

config/deny_gms_branch_unittests.lst

@@ -0,0 +1,59 @@
+# List of files for gms branches to delete
+# Use comment to allow list the file
+# Use curly braces to denote branch_id
+agree_bufgrid.tif
+agree_bufgrid_allo.tif
+agree_bufgrid_dist.tif
+agree_smogrid.tif
+agree_smogrid_allo.tif
+agree_smogrid_dist.tif
+catch_list_{}.txt
+coordFile_{}.txt
+crosswalk_table_{}.csv
+#demDerived_reaches_{}.dbf
+#demDerived_reaches_{}.prj
+#demDerived_reaches_{}.shp
+#demDerived_reaches_{}.shx
+#demDerived_reaches_split_{}.gpkg
+#demDerived_reaches_split_filtered_{}.gpkg
+demDerived_reaches_split_filtered_addedAttributes_crosswalked_{}.gpkg
+demDerived_reaches_split_points_{}.gpkg
+demDerived_streamPixels_{}.tif
+demDerived_streamPixels_ids_{}.tif
+demDerived_streamPixels_ids_{}_allo.tif
+demDerived_streamPixels_ids_{}_dist.tif
+dem_burned_{}.tif
+dem_burned_filled_{}.tif
+dem_lateral_thalweg_adj_{}.tif
+#dem_meters_{}.tif
+#dem_thalwegCond_{}.tif
+flowaccum_d8_burned_filled_{}.tif
+#flowdir_d8_burned_filled_{}.tif
+flowdir_d8_burned_filled_flows_{}.tif
+flows_grid_boolean_{}.tif
+flows_points_pixels_{}.gpkg
+gw_catchments_pixels_{}.tif
+#gw_catchments_reaches_{}.gpkg
+gw_catchments_reaches_{}.tif
+#gw_catchments_reaches_filtered_addedAttributes_{}.gpkg
+#gw_catchments_reaches_filtered_addedAttributes_{}.tif
+gw_catchments_reaches_filtered_addedAttributes_crosswalked_{}.gpkg
+headwaters_{}.tif
+#hydroTable_{}.csv
+idFile_{}.txt
+nld_rasterized_elev_{}.tif
+nwm_catchments_proj_subset_levelPaths_{}.gpkg
+nwm_subset_streams_levelPaths_{}.gpkg
+nwm_subset_streams_levelPaths_dissolved_headwaters_{}.gpkg
+rem_{}.tif
+#rem_zeroed_masked_{}.tif
+slopes_d8_dem_meters_{}.tif
+slopes_d8_dem_meters_masked_{}.tif
+sn_catchments_reaches_{}.tif
+src_{}.json
+src_base_{}.csv
+#src_full_crosswalked_{}.csv
+stage_{}.txt
+streamOrder_{}.tif
+treeFile_{}.txt
+#usgs_elev_table.csv

docs/CHANGELOG.md

@@ -1,6 +1,7 @@
 All notable changes to this project will be documented in this file.
 We follow the [Semantic Versioning 2.0.0](http://semver.org/) format.
 
+
 ## v4 (number pending) - 2022-07-27 - [PR #642](https://github.com/NOAA-OWP/inundation-mapping/pull/642)
 
 Fixes bug that causes [Errno2] No such file or directory error when running synthesize_test_cases.py if testing_versions folder doesn't exist (for example, after downloading test_cases from ESIP S3).
@@ -11,6 +12,8 @@ Fixes bug that causes [Errno2] No such file or directory error when running synt
 
 <br/><br/>
 
+=======
+
 ## v4.0.5.2 - 2022-07-25 - [PR #622](https://github.com/NOAA-OWP/inundation-mapping/pull/622)
 
 Updates to unit tests including a minor update for outputs and loading in .json parameter files.

unit_tests/README.md

@@ -32,10 +32,25 @@ One way is to use the incoming arg parser. Most py files, they include the code
 Start a docker container as you normally would for any development. ie) docker run --rm -it --name <a docker container name> -v /home/<your name>/projects/<folder path>/:/foss_fim {your docker image name}
 	- ie) docker run --rm -it --name Rob_GMS -v /home/rob_abcd/projects/dev/innudation-mapping/:/foss_fim -v /abcd_share/foss_fim/outputs/:/outputs -v /abcs_share/foss_fim/:/data fim_4:dev_20220208_8eba0ee
 
+For unit tests to work, you need to run the following (if not already in place).
+Notice a modified branch "deny_gms_branch_unittests.lst"  (special for unittests)
+
+Here are the params and args you need if you need to re-run unit and branch
+
+gms_run_unit.sh -n fim_unit_test_data_do_not_remove -u "02020005 02030201 05030104" -c /foss_fim/config/params_template.env -j 1 -d /foss_fim/config/deny_gms_unit_default.lst -o -s
+
+gms_run_branch.sh -n fim_unit_test_data_do_not_remove -c /foss_fim/config/params_template.env -j 1 -d /foss_fim/config/deny_gms_branch_unittests.lst -o -s
+
+**NOTICE: the deny file used for gms_run_branch... its a special one for unittests `deny_gms_branch_unittests.lst`.
+
+If you need to run inundation tests, fun the following:
+
+python3 foss_fim/tools/synthesize_test_cases.py -c DEV -e GMS -v fim_unit_test_data_do_not_remove -jh 1 -jb 1 -m /outputs/fim_unit_test_data_do_not_remove/alpha_test_metrics.csv -o
+
+If you want to test just one unit test, here is an example:
 At the root terminal window, run:  python ./foss_fim/unit_tests/gms/derive_level_paths_unittests.py  or python ./foss_fim/unit_tests/clip_vectors_to_wbd_unittests.py
 (replace with your own script and path name)
 
-
 ## Key Notes for creating new unit tests
 1) All test functions must start with the phrase "test_". That is how the unit test engine picks it up. The rest of the function name does not have to match the pattern of {function name being tested} but should. Further, the rest of the function name should say what the test is about, ie) _failed_input_path.  ie) test_{some_function_name_from_the_source_code_file}_failed_input_path. It is fine that the function names get very long (common in the industry).
 
@@ -55,18 +70,15 @@ At the root terminal window, run:  python ./foss_fim/unit_tests/gms/derive_level
 
 9) One py file = one "{original py file name}_unittests.py" file.
 
-10) Sometimes you may want to run a full successful "happy path" version through gms_run_by_unit.sh (or similar), to get all of the files you need in place to do your testing. However.. you will want to ensure that none of the outputs are being deleted during the test. One way to solve this is to put in an invalid value for the "-d" parameter (denylist). ie) Normally:  gms_run_unit.sh -n gms_example_unit_tests -u 05030104 -c /foss_fim/config/params_template.env -j 1 -d /foss_fim/config/deny_gms_unit_default.lst -o, but ours would be 
-gms_run_unit.sh -n gms_example_unit_tests -u 05030104 -c /foss_fim/config/params_template.env -j 1 -d no_list -o. 
-
+10) Sometimes you may want to run a full successful "happy path" version through gms_run_by_unit.sh (or similar), to get all of the files you need in place to do your testing. However.. you will want to ensure that none of the outputs are being deleted during the test. One way to solve this is to put in an invalid value for the "-d" parameter (denylist). ie) Normally:  gms_run_unit.sh -n fim_unit_test_data_do_not_remove -u 05030104 -c /foss_fim/config/params_template.env -j 1 -d /foss_fim/config/deny_gms_unit_default.lst -o, but ours would be 
+gms_run_unit.sh -n fim_unit_test_data_do_not_remove -u 05030104 -c /foss_fim/config/params_template.env -j 1 -d no_list -o. 
 
 ## Future Enhancements
 1) We can automate triggers on these files for things like checking triggers or an single global "run_all_unittest" script, but for now.. its one offs.
 
 2) Better output from the unit tests including verbosity output control
 
-3) We will upgrade it so you can pass in a params.json file to the call to give you even more flexibility by passing in your own params.json (think params_template.json idea)
-
-4) Over time, it is expected that python code files will be broken down to many functions inside the file. Currently, we tend to have one very large function in each code file which makes unit testing harder and less specific. Generally for each function in a python code file will result in at least one "happy path" unit test function. This might require having test unit test outputs, such as sample tif or small gpkg files in subfolders in the unit tests folder, but this remains to be seen. Note: Our first two files of derive_level_paths_unittests and clip_vectors_to_wbd_unittests are not complete as they do not yet test all output from a method.
+3) Over time, it is expected that python code files will be broken down to many functions inside the file. Currently, we tend to have one very large function in each code file which makes unit testing harder and less specific. Generally for each function in a python code file will result in at least one "happy path" unit test function. This might require having test unit test outputs, such as sample tif or small gpkg files in subfolders in the unit tests folder, but this remains to be seen. Note: Our first two files of derive_level_paths_unittests and clip_vectors_to_wbd_unittests are not complete as they do not yet test all output from a method.
 
 
 ## testing for failing conditions
@@ -80,4 +92,15 @@ An example is in unit_tests/gms/Derive_level_paths_unittests.py -> test_Derive_l
 
 We have almost no "assert"s yet, but most unit test usually have one or more "assert" test. See https://docs.python.org/3/library/unittest.html for more details.
 
+## Unit tests currently available
+python3 /foss_fim/unit_tests/gms/derive_level_paths_unittests.py
+python3 /foss_fim/unit_tests/tools/inundate_unittests.py
+python3 /foss_fim/unit_tests/tools/gms_tools/inundate_gms_unittests.py
+python3 /foss_fim/unit_tests/clip_vectors_to_wbd_unittests.py
+python3 /foss_fim/unit_tests/filter_catchments_and_add_attributes_unittests.py
+python3 /foss_fim/unit_tests/rating_curve_comparison_unittests.py
+python3 /foss_fim/unit_tests/shared_functions_unittests.py
+python3 /foss_fim/unit_tests/split_flows_unittests.py
+python3 /foss_fim/unit_tests/usgs_gage_crosswalk_unittests.py
+