Tuflow

TUFLOW SWMM Tutorial M04: Difference between revisions

Browse history interactively
← Older edit
Content deleted Content added
VisualWikitext
 
(116 intermediate revisions by 3 users not shown)
Line 1:
= Introduction =
This tutorial, Module 4 of the TUFLOW SWMM tutorial dataset will demonstrate TUFLOW's in-bulitbuilt event management functionality. This powerful functionality runs multiple event combinations (e.g. magnitude, duration, temporal patterns, climate change) using a single set of control files, rather than creating a new set of control files for every event simulation. ForThis example,tutorial withoutdemonstrates implementinghow event management acan projectbe requiringused simulationon ofa thesmaller followingscale, eventrunning combinationsonly would6 require 144 TCF and bc_dbase files:models.<br>
The real power of event management is seen when running a model of a much greater size. For example, without implementing event management, a project requiring simulation of the following event combinations would require 144 TCF and bc_dbase files:
* 6 x Event Magnitude (AEP): 01p, 02p, 05p, 10p, 20p, 50p
:* 86 x Event Duration (hour)Magnitude: 1h01%, 2h2%, 3h5%, 6h10%, 9h,20% 12h,and 18h,50% Annual Exceedance 24hProbability (AEP)
:*8 x Event Duration: 1, 2, 3, 6, 9, 12, 18 and 24 hour storm
* 3 x Climate Change Horizon: current climate, 2050, 2100
:*3 x Climate Change Planning Horizon: current climate, 2050, 2100
Using predefined naming conventions, with TUFLOW's event management functionality, all the events can be executed from a single TCF and bc_dbase file!
Using predefined naming conventions, with TUFLOW's event management functionality, all the events can be executed from a single TCF and bc_dbase file!
 
This feature of TUFLOW makes the management of the model files easier, ensures consistency between the simulations and improves project quality control by reducing the likelihood of human error associated with incorrect file referencing. <br>
 
TUFLOW SWMM Tutorial Module 4 builds from the model created in <u>[[TUFLOW_SWMM_Tutorial_M03|TUFLOW SWMM Tutorial Module 3]]</u>. The completed TUFLOW SWMM Module 3 model is provided in the '''''TUFLOW_SWMM_Module_04\TUFLOW''''' folder of the download dataset as the starting point for this tutorial. If you are not already familiar with TUFLOW SWMM linking, we recommend first completing ModuleTUFLOW SWMM Modules 1, 2 and 3 beforeprior to starting this tutorial. <br>
 
= Simulation Control Files =
TUFLOW's automated event management operates by introducing wildcard referencing into the boundary condition database.<br>
A new control file, called the "'TUFLOW Event File"' (TEF) is added. It lists all event titles whichthat are applied to the wildwildcard charatctercharacter in the boundary condition database. It also lists unique commands to associate with the specific events.<br>
Models are executed using a batch file, with the event specifionspecifically listed as a variable in the batch file syntax.
 
This tutorial example will upgrade our single simulation model to run six event simulations. The simulations will cater for three Annual Exceedance Probability (AEP) event magnitudes: 5%, 2% and 1% AEP. In Average Recurrence Interval (ARI) terminology, this translates to the 1 in 20, 50 and 100 year ARI events. These three design events will be simulated for two event storm durations: 1 and 2 hours.<br>
 
This tutorial example will upgrade our single simulation model to run six (six) event simulations. The simulations will cater for three (3) Annual Exceedance Probability (AEP) event magnitudes: 5%, 2% and 1% AEP (in Average Recurrance Interval (ARI) terminology, this translates to the 1 in 20, 50 and 100 year ARI events). The design storms will be simulated for two (2) event storm durations: 1 and 2 hour.
=== TUFLOW Boundary Condition Database (bc_dbase) ===
The first task when updatingupgrading a model withto include Event Management functionality is to updatedecide theon bc_dbase.csva withnaming eventconvention magnitudefor andthe event duration wildcard placeholders:characters to be used in the bc_dbase.csv.
*Event Magnitude: '''AEP'''
*Event Duration: '''DUR'''
Update the bc_dbase with event magnitude and event duration wildcard characters.
<ol>
<li>Navigate to the '''TUFLOW_SWMM_Module_04\Tutorial_Data''' folder. Copy and paste the following files to the '''TUFLOW_SWMM_Module_04\TUFLOW\bc_dbase''' folder. These files contain the inflows and rainfall for each event magnitude and duration.
<li>We will use the following wildcard characters to represent the Events in our model:
:*'''TS04_inflow_05p1hr.csv'''
<li>Event Magnitude = AEP
:*'''TS04_inflow_05p2hr.csv'''
 
:*'''TS04_inflow_02p1hr.csv'''
<li>Event Duration = DUR
:*'''TS04_inflow_02p2hr.csv'''
 
:*'''TS04_inflow_01p1hr.csv'''
<li>Navigate to the '''TUFLOW_SWMM_Module_04\Tutorial_Data''' folder. Copy and paste the following files to the ''' TUFLOW_SWMM_Module_04\TUFLOW\bc_dbase''' folder. These files contains the inflows for each event magnitude and duration.
:*'''TS04_inflow_01p2hr.csv'''
:*TS04_inflow_05p1hr.csv
:*'''TS04_rainfall_05p1hr.csv'''
:*TS04_inflow_05p2hr.csv
:*'''TS04_rainfall_05p2hr.csv'''
:*TS04_inflow_02p1hr.csv
:*'''TS04_rainfall_02p1hr.csv'''
:*TS04_inflow_02p2hr.csv
:*'''TS04_rainfall_02p2hr.csv'''
:*TS04_inflow_01p1hr.csv
:*'''TS04_rainfall_01p1hr.csv'''
:*TS04_inflow_01p2hr.csv
:*'''TS04_rainfall_01p2hr.csv''' <br>
:*TS04_rainfall_05p1hr.csv
The files with 'inflow' in their name are flow hydrograph boundary conditions sourced from an external hydrology model, such as HEC-HMS, RORB, URBS, WBNM etc. <br>
:*TS04_rainfall_05p2hr.csv
The files with 'rainfall' in their name are rainfall depth hyetographs used to define rainfall timeseries in the SWMM urban hydrology section of the model.
:*TS04_rainfall_02p1hr.csv
:*TS04_rainfall_02p2hr.csv
:*TS04_rainfall_01p1hr.csv
:*TS04_rainfall_01p2hr.csv <br>
The files with 'inflow' in their name are flow hydrograph boundary conditions sourced from an external hydrology model, such as HEC-HMS, RORB, URBS, WBNM etc. The files with 'rainfall' in their name are rainfall depth hyetographs used to define the rainfall timeseries in the SWMM urban hydrology section of the model.
 
<li>Save a copy of the '''bc_dbasebc_dbase_TS03_001.csv''' as '''bc_dbase_TS04_001.csv'''.
<li>Open the file and make the following updates to the 'Source' column. The 'AEP' and 'DUR' wildcards must be between the special character '~' to function as a variable. Note, columns E to I are to remain blank. The final '''bc_dbase_TS04_001.csv''' should look like this: <br>
<br>
[[File:SWMM_M04_Event_bcdbase_001SWMM_M04_event_bcdbase_01a.JPG||500pxpng]]<br>
<br>
<li>Save '''bc_dbase_TS04_001.csv'''.
<br><br>
{{Video|name=Animation_TS4_Sim_Control_01b.mp4|width=1236}}
<br>
</ol>
 
Line 52 ⟶ 54:
<ol>
<li>Navigate to the '''TUFLOW_SWMM_Module_04\Tutorial_Data''' folder. Copy the '''TS04_Event_File.tef''' to the '''TUFLOW_SWMM_Module_04\TUFLOW\runs''' folder.
<li>Open the TEF in a textText editorEditor (Notepad++ is recommended). There are three event magnitudes and two event durations listed:
:*Event Magnitudes: 5% , 2% and 1% AEP
:*Event Durations: 11hr and 2 hour 2hrs
<li>The event specific commands are contained between the <font color="blue"><tt>Define Event</tt></font> and <font color="blue"><tt>End Define</tt></font> commands.
<li>The <font color="blue"><tt>BC Event Source</tt></font> command is linked to the bc_dbase file ('''bc_dbase_TS04_001.csv'''). In the below examples, any occurrence of '~AEP~' in '''bc_dbase_TS04_001.csv''' will be replaced with '05p' and any occurrence of '~DUR~' in '''bc_dbase_TS04_001.csv''' will be replaced with '1hr'. Also, when the 1 hour event is called the simulation, runthe end time will be set to 2 hours (instead of 3 hours, as per the other simulations). <br>
<font color="green"><tt>! EVENT MAGNITUDES</tt></font><br>
<font color="blue"><tt>Define Event</tt></font> <font color="red"><tt> == </tt></font> <font color="black"><tt>05p</tt></font>
Line 63 ⟶ 65:
<font color="green"><tt>! EVENT DURATIONS</tt></font><br>
<font color="blue"><tt>Define Event</tt></font> <font color="red"><tt> == </tt></font> <font color="black"><tt>1hr</tt></font>
<ol><font color="blue"><tt> BC Event Source </tt></font> <font color="red"><tt>==</tt></font> <font color="black"><tt> ~DUR~ | 1hr</tt></font><br>
<font color="blue"><tt> End Time </tt></font> <font color="red"><tt>==</tt></font> <font color="black"><tt> 2</tt></font>
</ol><font color="blue"><tt>End Define</tt></font>
<li> When TUFLOW runs a simulation, it reads all of the defined events in the TEF, though only uses the event variables specified in the batch file. We will create a compatible batch file shortly.<br>
<br><br>
{{Video|name=Animation_TS4_Sim_Control_02c.mp4|width=1236}}
<br>
</ol>
 
=== TUFLOW Control File (TCF) ===
<ol>
<li>Save a copy ofOpen '''TS03_5m_001.tcf''' in a Text Editor (Notepad++ is recommended). Save the file as '''TS04_5m_~e1~_~e2~_001.tcf''' in the '''TUFLOW_SWMM_Module_04\TUFLOW\runs''' folder.
<li>Make the following reference update:<br>
<li>Open the file '''TS04_5m_~e1~_~e2~_001.tcf''' in a text editor.
<font color="blue"><tt>BC Database </tt></font> <font color="red"><tt>== </tt></font> <font color="black"><tt>..\bc_dbase\bc_dbase_TS04_001.csv</tt></font> <font color="green"><tt> ! Reference the Boundary Conditions Database</tt></font> <br>
<li>Update the reference to the bc_dbase:<br>
<li>Add the following commands to reference the TEF at the end of the TCF: <br>
<font color="blue"><tt>BC Database </tt></font> <font color="red"><tt>== </tt></font> <font color="black"><tt> ..\bc_dbase\bc_dbase_TS04_001.csv </tt></font> <font color="green"><tt> ! Reference the Boundary Conditions Database</tt></font> <br>
<li>Add the following command to reference the TEF at the end of the TCF: <br>
<font color="green"><tt>! EVENTS</tt></font> <br>
<font color="blue"><tt>Event File </tt></font> <font color="red"><tt>== </tt></font> <font color="black"><tt> TS04_Event_File.tef </tt></font> <font color="green"><tt> ! Reference the TUFLOW Event File</tt></font> <br>
The location of this command in the TCF is important. If there are any commands associated with an event which overwrite other existing commands (such as <font color="blue"><tt>End Time</tt></font> <font color="red"><tt>==</tt></font>), the TEF file needs to be processed last for it to take precedence over other commands fulfilling the same purpose. Referencing this file in the last line in the TCF ensures this order of operation occurs as intended.<br>
<li>Save the TCF.
</olbr><br>
{{Video|name=Animation_TS4_Sim_Control_03c.mp4|width=1236}}
<br>
</ol>
 
= Running the Simulation =
Line 87 ⟶ 94:
 
<li>Update the batch file to reference the '''TS04_5m_~e1~_~e2~_001.tcf''' and include event switches (-e1, -e2) defining event magnitude and event duration:<br>
<font color="blue"><tt>'''set'''</tt></font> <font color="black"><tt>exe</tt></font><font color="red"><tt>=</tt></font><font color="black"><tt>"..\..\..\exe\2023-03-AAAF\TUFLOW_iSP_w64.exe"</tt></font> <br>
<font color="blue"><tt>'''set'''</tt></font> <font color="black"><tt>run</tt></font><font color="red"><tt>=</tt></font><font color="black"><tt>start "TUFLOW" /wait</tt></font> <font color="orange"><tt> %exe%</tt></font> <font color="black"><tt> -b</tt></font> <br>
<font color="orange"><tt>%run% </tt></font> <font color="black"><tt> -e1 05p -e2 1hr TS04_5m_~e1~_~e2~_001.tcf </tt></font> <br>
Line 95 ⟶ 102:
<font color="orange"><tt>%run% </tt></font> <font color="black"><tt> -e1 01p -e2 1hr TS04_5m_~e1~_~e2~_001.tcf </tt></font> <br>
<font color="orange"><tt>%run% </tt></font> <font color="black"><tt> -e1 01p -e2 2hr TS04_5m_~e1~_~e2~_001.tcf </tt></font> <br>
<li>Save the batch file and double click it in fileWindows explorerFile Explorer to run the simulation.<br>
If the model simulation doesn't run, here is a link to some common <u>[[Tutorial_Troubleshooting_QGIS | Trouble Shooting]]</u> advice.
</ol>
<br><br>
{{Video|name=Animation_TS4_Run_Sim_01c.mp4|width=1236}}
<br>
</ol>
 
== Troubleshooting ==
Did your TUFLOW SWMM model fail to run successfully? If so, here is a link to a troubleshooting guide: <u>[[TUFLOW_SWMM_Troubleshooting | TUFLOW SWMM Troubleshooting]]</u>.<br>
 
=Advanced Looping Batch File (Optional)=
= Check Files =
Looping batch files are a powerful tool to enable high volume simulation management with the least amount syntax, and subsequently manual effort. For example, the following looped batch file executed 1440 simulation using 15 lines of syntax!
While the model is running, review the added features are specified correctly:
<olbr><br>
[[File:SWMM_M04_advanced_looping_batch_01c.png]]<br>
<li>Open the '''M09_5m_05p_1hr_001.tlf''' from the '''Module_09\TUFLOW\runs\log''' folder in a text editor.
<li>The name of the events replaced the placeholders in the TCF name.
<li>Confirm the specified events and event sources:<br>
{{Video|name=Animation_M09_Check_01c.mp4|width=928}}
<br>
</ol>
 
=Results=
 
= Conclusion =
:*Multiple events were set up in the TUFLOW Event File (TEF) with define event blocks.
:*TUFLOW Log Files (TLF) were reviewed.
:*Multiple event results were inspected using the TUFLOW Viewer.
<br>
 
We will convert our sequential formatted batch file to a looped format:
=Looping Batch File (Optional)=
Looping batch file is a powerful tool running large number of simulations with the least amount of batch file text:
<ol>
<li>Save a copy of '''_run_M09_HPC_run_TS04_HPC.bat''' as '''_run_M09_HPC_loop_run_TS04_HPC_loop.bat''' in the '''Module_09TUFLOW_SWMM_Module_04\TUFLOW\runs''' folder.
<li>Add the following commands. Indentation is important in batch files, insert tabs to match the below spacing:<br>
<font color="blue"><tt>echo </tt></font> <font color="black"><tt>off</tt></font><br>
<font color="blue"><tt>set </tt></font> <font color="black"><tt>exe</tt></font><font color="red"><tt>=</tt></font><font color="black"><tt>"..\..\..\exe\2023-03-AAAF\TUFLOW_iSP_w64.exe"</tt></font><br>
<font color="blue"><tt>set </tt></font> <font color="black"><tt>run</tt></font><font color="red"><tt>=</tt></font><font color="black"><tt>start "TUFLOW" /wait </tt></font><font color="orange"><tt>%exe%</tt></font><font color="black"><tt> -b</tt></font><br>
<br>
Line 133 ⟶ 131:
<font color="blue"><tt>for </tt></font><font color="orange"><tt>%%a</tt></font><font color="blue"><tt> in </tt></font><font color="black"><tt>(</tt></font><font color="orange"><tt>%AEP%</tt></font><font color="black"><tt>)</tt></font><font color="blue"><tt> do </tt></font><font color="black"><tt>(</tt></font><br>
<ol><font color="blue"><tt> for </tt></font><font color="orange"><tt>%%b</tt></font><font color="blue"><tt> in </tt></font><font color="black"><tt>(</tt></font><font color="orange"><tt>%DUR%</tt></font><font color="black"><tt>)</tt></font><font color="blue"><tt> do </tt></font><font color="black"><tt>( </tt></font><br>
<ol><ol><font color="blue"><tt> echo </tt></font><font color="black"><tt>Running </tt></font><font color="orange"><tt>%%a %%b </tt></font><br>
<font color="orange"><tt> %run% </tt></font><font color="black"><tt>-e1 </tt></font><font color="orange"><tt>%%a</tt></font><font color="black"><tt> -e2 </tt></font><font color="orange"><tt>%%b </tt></font><font color="black"><tt>M09_5m_TS04_5m_~e1~_~e2~_001.tcf</tt></font> <br>
</ol></ol> )<br>
</ol>)<br>
'''Note:''' The 'AEP' and 'DUR' are user defined batch file variables. Projects may use different variable names. The 'AEP' refers to an event magnitude (Annual Exceedance Probability) and 'DUR' refers to an event duration.
 
<li>Save the batch file and double click it in file explorer to run the simulation.
<li>Save the batch file and double click it in Windows File Explorer to run the simulation.
</ol><br>
If the model simulation doesn't run, here is a link to some common <u>[[Tutorial_Troubleshooting_QGIS | Trouble Shooting]]</u> advice.
<br><br>
{{Video|name=Animation_TS4_Run_Sim_02b.mp4|width=1236}}
<br>
</ol>
 
= Check and Result Files =
When the TUFLOW models were executed via the batch file, the names of check and result files should have been updated to include the event variables from the simulation. For example:
 
{| class="wikitable" width="60%" style="text-align:center"
!style="background-color:#005581; font-weight:bold; color:white;"| TCF
!style="background-color:#005581; font-weight:bold; color:white;"| Event 1 (-e1)
!style="background-color:#005581; font-weight:bold; color:white;"| Event 2 (-e2)
!style="background-color:#005581; font-weight:bold; color:white;"| Check and Result filename
|-
|rowspan="6"|TS04_5m_<font color="red">'''~e1~_~e2~'''</font>_001.tcf
| 05p || 1hr || TS04_5m_<font color="red">'''05p_1hr'''</font>_001
|-
| 05p || 2hr || TS04_5m_<font color="red">'''05p_2hr'''</font>_001
|-
| 02p || 1hr || TS04_5m_<font color="red">'''02p_1hr'''</font>_001
|-
| 02p || 2hr || TS04_5m_<font color="red">'''02p_2hr'''</font>_001
|-
| 01p || 1hr || TS04_5m_<font color="red">'''01p_1hr'''</font>_001
|-
| 01p || 2hr || TS04_5m_<font color="red">'''01p_2hr'''</font>_001
|}
 
==Check Sucessful Completion of Multiple Simulations==
Open '''_TUFLOW Simulations.log''' from the '''TUFLOW_SWMM_Module_04\TUFLOW\runs''' folder. Confirm 'Finished' is tagged next to the entry for each of the six simulations we executed. This log confirms the model simulation was completed.<br>
<br>
{{Video|name=Animation_TS4_Results_01b.mp4|width=1350}}
<br>
 
==Viewing Multiple Simulation Results==
===Load Results===
Open the simulation results in QGIS. When loading multiple simulations which include 1D results (all with the same object IDs), a two step load workflow is recommended to avoid filling the QGIS layer panel with duplicate 1D object entries.
<ol>
<li> Open TUFLOW Viewer from the TUFLOW Plugin toolbar.<br>
<br>
[[File:tuflow_plugin_tuflow_viewer.png]]<br>
<br>
<li> Open 1D and 2D results: Select File > Load Results. Navigate to the '''TUFLOW\runs''' folder and select '''TS04_5m_~e1~_~e2~_001.tcf'''.
<li> When asked which simulation to load, select '''TS04_5m_01p_1hr_001'''.
<li> When asked 'Do you want to open result GIS layer?', select 'Yes'.
<li> After '''TS04_5m_01p_1hr_001''' has loaded, open the remaining result files:
* Select File > Load Results. Navigate to the '''TUFLOW\runs''' folder and select '''TS04_5m_~e1~_~e2~_001.tcf'''.
* When asked which simulation to load, select all remaining results (except '''TS04_5m_01p_1hr_001''').
* When asked 'Do you want to open result GIS layer?', select 'No'. The results will still be loaded, though will all be associated with the GIS objects loaded for the '''TS04_5m_01p_1hr_001''' simulation.
<li> In the QGIS Layers Panel, move the GIS vector (line and point) layers to the top of the list. This will ensure these layers are displayed above all mesh result layers in the project.
<br><br>
{{Video|name=Animation_TS4_Results_02c.mp4|width=1350}}
<br>
</ol>
 
===Mesh Results (Map Output)===
Inspect the results for all events.
<ol>
<li>Zoom in to the pipe network area.
<li>Click on the scalar dataset 'Depth' under the 'Result Type' in the TUFLOW Viewer 'Map Outputs' list. Toggle on the maximum depth output by clicking the [[File: max_output_icon.png]] symbol.
<li>Click through the results in the 'Open Results' list to observe the different max depth values.
<br><br>
{{Video|name=Animation_TS4_Results_03c.mp4|width=1350}}
<br>
</ol>
 
===Vector Results (Pipe Network Output)===
<ol>
<li>In the QGIS Layers Panel, right click '''TS04_5m_01p_1hr_001_swmm_ts_L''', and select 'Show Labels'.
<li>Use the 'Select Features' tool to highlight 'Pipe 3'.
<li>In the 'Open Results' list, select all 1hr duration event results (hold Ctrl to select multiple).
<li>Click on a scalar dataset 'Flow' under the 'Result Type' in the TUFLOW Viewer 'Time Series' list.
<li>Try multiple combinations of the simulation results to observe how the different events impacted flow through 'Pipe 3'.
<li>Click the 'Refresh Current Plot' [[File: Refreshplotblack.png || 20px]] button to update the plot window after making new selections.
<li>Click the 'Clear Current Plot' [[File: QGIS_Plugin_Icon_ClearPlot.png || 20px]] button to remove all data from the plot window if needed prior to making a new selection.
<br><br>
{{Video|name=Animation_TS4_Results_04c.mp4|width=1350}}
<br>
</ol>
 
= Conclusion =
*Automated Event Management functionality was added to the TUFLOW SWMM model.
*Multiple event simulations were run in sequence from a batch file.
*Creation of looped batch file was demonstrated for the efficient execution of bulk simulations.
*Multiple event simulation results were inspected using the QGIS TUFLOW Viewer.
*For further training opportunities see <u>[https://tuflow.com/training/training-course-catalogue/ TUFLOW Training Catalogue]</u> and/or contact <u>[mailto:training@tuflow.com training@tuflow.com]</u>. <br>
<br>
 
=Other TUFLOW SWMM Tutorials=
*<u>[[TUFLOW_SWMM_Tutorial_M01 | TUFLOW SWMM Module 1]]</u> - 1D SWMM Culverts
*<u>[[TUFLOW_SWMM_Tutorial_M02 | TUFLOW SWMM Module 2]]</u> - 1D SWMM Pipe Network / 2D TUFLOW Direct Rainfall Hydrology
*<u>[[TUFLOW_SWMM_Tutorial_M03 | TUFLOW SWMM Module 3]]</u> - 1D SWMM Pipe Network / 1D SWMM Urban Hydrology
*<u>[[TUFLOW_SWMM_Tutorial_M04 | TUFLOW SWMM Module 4]]</u> - 1D SWMM Pipe Network / 1D SWMM Urban Hydrology: Executing multiple different event simulations from a single model control file.
*<u>[[XPSWMM_to_TUFLOW-SWMM | XPSWMM to TUFLOW SWMM]]</u> - How to convert an XPSWMM model to TUFLOW SWMM.
<br>
{{Tips Navigation
Retrieved from "https://wiki.tuflow.com/TUFLOW_SWMM_Tutorial_M04"