Thalweg Profile Tool and lateral Thalweg Adjustment Threshold

[BrianAvant]

Adding a thalweg profile tool that checks for significant elevation drops along thalweg. Also setting lateral thalweg adjustment threshold in hydroconditioning. Resolves #408, Resolves #18, and Resolves #409

Additions

• thalweg_drop_check.py checks the elevation along the thalweg for each stream path downstream of MS headwaters within a HUC8.

  Outputs

  • Plots for each HUC where each plot shows the thalweg elevation profile for a MS headwater stream path. Elevation grids compared include the orignal DEM, DEM with lateral thalweg adjustment, and the final thalweg adjusted DEM, all in meters.
  • Points layer containing the location of all thalweg cells that drop below a threshold of 3 meters and additionally checks for any rise in elevation along the thalweg greater than 0.0 meters.

  Tool Args

  • fim_dir: directory of FIM outputs
  • output_dir: directory of tool outputs
  • stream_type: stream layer to define thalweg. Options are derived and 'burnline'. Using the 'burnline' option involves extra steps to add verticies for every cell along the thalweg. The 'derived' option is recommended for use in post-processing. The 'burnline' option is only useful for updating thalweg cell values during the hydroconditioning in run_by_unit.sh and has not been fully tested.
  • point_density: elevation sample density. Options are 'all_points' or 'midpoints'.
  • th: significant elevation drop threshold in meters.
  • j: number of jobs to run in parallel

  Example call

  python foss_fim/tools/thalweg_drop_check.py -fim_dir /data/outputs/test -output_dir /data/tools/thalweg_profile_comparison/single_pixel_hucs_gpd -stream_type derived -point_density all_points -th 3.0 -j 12

  Plots

  

Removals

• Removing dissolveLinks arg from clip_vectors_to_wbd.py.

Changes

• Cleaned up code in split_flows.py to make it more readable.
• reduce_nhd_stream_density.py and adjust_headwater_streams.py were refactored to limit MS headwater point in agg_nhd_headwaters_adj.gpkg.
• Fixed a bug in adjust_thalweg_lateral.py where change in elevation between thalweg and lateral elevations where limited by threshold and updated this threshold to 3 meters.
• Improvement of log feedback in aggregate_vector_inputs.py to identify anomalies when preprocessing NHDPlus HR stream network and updated attributes to match new nws_lid.gpkg layer.
• Updated output_cleanup.py -p whitelist to include layers for thalweg_drop_check.py tool. (dem_lateral_thalweg_adj.tif, dem_thalwegCond.tif, dem_meters.tif, demDerived_reaches_split.gpkg, nhd_headwater_points_subset.gpkg, wbd.gpkg, NHDPlusBurnLineEvent_subset.gpkg)

Testing

Evaluation metrics were calculated for this feature branch as part of the standard practice but most of the sites that show single pixel inundation are not included in the list of sites with evaluation data. Sites flagged as having single pixel inundation were observed from a qualitative review of the CatFIM outputs as well as through feedback from the field offices. The list of flagged sites from previous CatFIM outputs was used to identify and categorize the source of single pixel inundation. Compared to FIM 3.0.17.1, there was no change or a slight improvement in aggregate CSI scores for NWS and USGS sites except for NWS-moderate and NWS-action stages. BLE aggregate CSI shows a regression in the 100yr and 500yr flood events.

The initial hypothesis was that large drops in elevation, introduced in the hydroconditioning process, were the main cause of the observed single pixel inundation. I created a tool to indentify the thalweg elevation profile for each individual path originating at the MS headwater point (always a NWS LID). This tool was then used to identify elevation drops in the thalweg at or below 5 meters initially but then lowered to 3 meters after observing the outputs.

Screenshots

NWS LID SORI2; Action Stage; HUC 07120004; NCRFC
FIM v3.0.19.0/dev-single-pixel-inundation:

Closer look:

Issues that still remain:

• Disagreements between NWM and derived NHDPlus HR stream networks

1. FIM 3 input stream network uses the ToNode/FromNode and LevelPath attributes to traverse the HR NHDPlus stream network in an attempt to match the density of the NWM stream network used for producing flows. This does not do any vetting of the stream path to ensure that there are no gaps, artificial paths, and other considerations used to convert the NHDPlus medium resolution stream network to the NWM routlink file. It is observed in many cases where the subset HR network will follow an adjacent artificial path, especially below dams. Irrigation lines sometimes cut through multiple catchments. The NHDPlus has attributes to remove these types of segments from the dataset (FType/FCode) but the dataset is still in progress and may not be mature enough to rely on these attributes for our purposes.

2. FIM 3 input stream network does not include braided stream segments. Differences in catchment delineation, braided segment selection of HR vs NWM, and major changes in the stream network geometry make mapping and crosswalking challenging in these areas.

   Instead of trying to resolve these types of issues within the FIM workflow, it would probably be better to use the same starting network (spatial resolution and density) for both models to derive flow and inundation libraries.

• Distributed networks

  Besides the issues mentioned in the previous point, inundation mapping at highly braided networks usually underperforms due to the simplified way that NWM routes flow through these areas. The current NWM v2.1 configuration selects a single segment to receive 100% of the upstream flow in distributed sections of the network. Alternate segments can still have some flow accumulating from within their own catchments. This simplification is reasonable for flow routing but will likely not adequately represent the needs of FIM.

  NWS LID LMDN5 Action Stage - HUC 13020102
  

• Steep Terrain

  • FIM 3 performs poorly in some areas where there is steep terrain and narrow canyon walls. This is in part likely due to resolution limitations of the input data.
  • True elevation drops below dams can be greater than 5 meters and several cases are observed where this causes single pixel inundation directly below the dam and even sections much further downstream. We could look at easing the elevation gradient in these areas but this could have undesirable side effects. One task we could do to help with this specific issue would be to enforce a break in the DEM derived stream network at dams. Doing this would ensure the catchments didn't contain a mix of higher and lower elevations which compromises the REM.

• Very small streams

  • In some cases, limitations in the resolution of input data prevent adequate representation of inundation. We will need higher resolution elevation data to resolve this case.

• Missing/Bad input data

  • It seems that several sites around the Mexico/US and Canadian/US borders perform quite poorly. It is possible that this is related to partial or incomplete data in these areas. There are also a few areas well within the continental U.S. that have similar issues. There is a related issue we have labeled as spotty inundation around the Great Lakes. Salt Lake City is another example. There are several issues going on that make it challenging to untangle.
    Salt Lake City, NWS LID CCSU1 Action Stage - HUC 16020204
    

• Misc
  There are several other cases that would need further investigation to resolve. There may be commonality in the root cause of many of the issues related to single pixel inundation but this requires more analysis and I am not sure at the moment this would be the best return on investment.

  • Stream below dam follows correct path but inundation is mapped primarily on diversion channel (MS):
    NWS LID FWLC2 Action Stage - HUC 11020005
    

  • Streams with high sinuosity: Limitations in the vector representation of stream networks could also partially explain single pixel inundation in some areas as well.
    NWS LID LPSC2 Action Stage - HUC 14080105
    

  • Sinks adjacent to stream: This was partially resolved with setting a elevation threshold on the lateral thalweg adjustment but there are still some places where single pixel inundation remains. In general, there should probably be a "rule" in FIM 3 that limits or eliminates situations where there is inundation within a catchment that does not connect back to the stream. We can probably resolve this in the hydroconditioning.
    NWS LID GATU1 Action Stage - HUC 16020102
    

  • Stream confluences: This is something that we were seeing in FIM 2 and would probably be wise to reach out the the ESRI Arc Hydro team. At the confluence of two tributaries, significant differences (> 5 meters) in thalweg elevation cause an immediate drop to the thalweg path of the higher stream. My initial hypothesis assumed this would explain the majority of the single pixel inundation issues but it turns out that there are only a few instances in FIM 3.

  • False continuity in derived stream network: Sometimes nearby streams are joined together during the derivation of the FIM stream network. In this example near Scottsdale, AZ, a sharp, unnatural HUC 8 boundary intersects a MS headwater stream segment and it is rerouted to a nearby stream,
    NWS LID IDXA3 Action Stage - HUC 15060106
    

Notes

thalweg_drop_check.py needs to be updated so that it checks HUCs with no MS headwater points. The HUC8 intersection points would probably be the best choice for including these HUC profiles.

[BrianAvant]

@BradfordBates-NOAA I might get you to run this tool to test it. I have moved some things around and would like to make sure I didn't break something in the process.