Difference between revisions of "ARR 2019 Ensembles Processing"

From Tuflow
Jump to navigation Jump to search
(40 intermediate revisions by 3 users not shown)
Line 1: Line 1:
<span style="color:#FF0000">
 
<font size = 18>Page Under Construction</font>
 
</span>
 
 
This page summarizes recommended approach for processing a large amount of ARR 2016 ensembles.
 
 
==Introduction==
 
==Introduction==
The ARR 2016 guidelines implemented ten temporal patterns which are to be applied for each annual exceedance probability (AEP) and duration. This multiplies the number of simulations by ten and requires additional processing in comparison with ARR 1987. The results files can get very large, the creation of the processing tools can lead to tedious work and the actual processing can be time intensive.
+
The Australian Rainfall and Runoff (ARR) 2016 guideline requires the use of ten temporal patterns for each design Annual Exceedance Probability (AEP) and storm duration combination. This multiplies the required number of design simulations by ten and requires additional processing compared to the previous approach recommended in ARR 1987. As a consequence, the model result folders can get very large and the post processing of the results can be time intensive. The below page provides recommendations to minimise the effort and footprint of running TUFLOW for using the latest ARR 2016 guideline ensemble approach.
 
==Choice of Output==
 
==Choice of Output==
As the model results might get large the preferred method to process ensembles is to use .flt grid output format to export the maximums. FLT files are the ESRI binary (float) version of the ASC files and their size is about 1/5 of a ASC file. All binary files can also be processed faster by TUFLOW utilities.
+
As the model result folder can get large in size with the additional simulations, the preferred maximum grid output format is FLT, which can be done directly from the TUFLOW simulation. FLT files are ESRI binary (float) version of the ASC files, their size is about 1/5 of an ASC file. Binary files can also be processed faster by TUFLOW utilities.<br>
 +
Commands in the TCF:<br>
 +
:<font color="blue"><tt>Map Output Format </tt></font> <font color="red"><tt>== </tt></font><tt>FLT</tt><br>
 +
:<font color="blue"><tt>Map Output Date Types </tt></font> <font color="red"><tt>== </tt></font><tt>h, d, V</tt><br>
 +
:<font color="blue"><tt>FLT Map Output Interval </tt></font> <font color="red"><tt>== </tt></font><tt>0</tt><br>
 +
If using only time-series outputs (xmdf, dat) an extra step will be required to create these maximum grids (FLT or ASC). This can be done using TUFLOW_to_GIS utility.
 +
 
 +
==Median==
 +
Use <u>[https://wiki.tuflow.com/index.php?title=ASC_to_ASC#.22-statMedian.22 ASC_to_ASC utility (-statMedian switch)]</u> to create the median result of required output (water level, depth, velocity, hazard, ...) for each AEP and storm duration combination.
 +
 
 +
==Maximums==
 +
Use <u>[https://wiki.tuflow.com/index.php?title=ASC_to_ASC#Maximum ASC_to_ASC utility (-max switch)]</u> to create the maximum envelope of the median results for each AEP.
 +
==Processing Batch File Tips==
 +
*The location of the batch file is recommended to be in the same folder as the result files. This will enable the use of relative path for the input and output files to make the command syntax shorter.<br>
 +
:<u>Example:</u><br>
 +
:This method would be preferable:<br>
 +
:<tt>"C:\TUFLOW\Utilities\asc_to_asc_w64.exe" -b -out ARR_Max.flt -max ARR_060m.flt ARR_120m.flt</tt><br>
 +
:Instead of this method:<br>
 +
:<tt>"C:\TUFLOW\Utilities\asc_to_asc_w64.exe" -b -out "C:\Projects\Project_1024\Processing\ARR_Max.flt" -max "C:\Projects\Project_1024\Processing\ARR_060m.flt" "C:\Projects\Project_1024\Processing\ARR_120m.flt"</tt><br>
 +
*A wild card (*) can be used to avoid typing all the file names. Instead ASC_to_ASC utility will automatically read all the files with the same start and end of the filename.<br>
 +
:<u>Example:</u><br>
 +
:This method would be preferable:<br>
 +
:<tt>"C:\TUFLOW\Utilities\asc_to_asc_w64.exe" -b -out ARR_Max.flt -max ARR_*.flt</tt><br>
 +
:Instead of this method:<br>
 +
:<tt>"C:\TUFLOW\Utilities\asc_to_asc_w64.exe" -b -out ARR_Max.flt -max ARR_060m.flt ARR_120m.flt ARR_180m.flt ARR_240m.flt ARR_300m.flt ARR_360m.flt</tt><br>
 +
*Looping through variables in a batch file can also be used to make the processing more efficient. Example of such batch file is below:<br>
 +
 
 +
<pre>
 +
@echo off
 +
 
 +
:: This sets the variables as local, so you can use another batch file with A and B variables
 +
SetLocal
 +
 
 +
:: set path to ASC_to_ASC Utility
 +
set ASC_to_ASC="C:\TUFLOW\Utilities\asc_to_asc_w64.exe"
 +
 
 +
:: set up variables
 +
set A=060m 120m 180m 240m 300m 360m
 +
set B=tp01 tp02 tp03 tp04 tp05 tp06 tp07 tp08 tp09 tp10
 +
 
 +
:: Loop Through
 +
FOR %%a in (%A%) do (
 +
    FOR %%b in (%B%) DO (
 +
%ASC_to_ASC% -b -statMedian ARR_%%a_%%b_001.flt
 +
    )
 +
)
 +
pause
 +
</pre>

Revision as of 16:41, 5 March 2021

Introduction

The Australian Rainfall and Runoff (ARR) 2016 guideline requires the use of ten temporal patterns for each design Annual Exceedance Probability (AEP) and storm duration combination. This multiplies the required number of design simulations by ten and requires additional processing compared to the previous approach recommended in ARR 1987. As a consequence, the model result folders can get very large and the post processing of the results can be time intensive. The below page provides recommendations to minimise the effort and footprint of running TUFLOW for using the latest ARR 2016 guideline ensemble approach.

Choice of Output

As the model result folder can get large in size with the additional simulations, the preferred maximum grid output format is FLT, which can be done directly from the TUFLOW simulation. FLT files are ESRI binary (float) version of the ASC files, their size is about 1/5 of an ASC file. Binary files can also be processed faster by TUFLOW utilities.
Commands in the TCF:

Map Output Format == FLT
Map Output Date Types == h, d, V
FLT Map Output Interval == 0

If using only time-series outputs (xmdf, dat) an extra step will be required to create these maximum grids (FLT or ASC). This can be done using TUFLOW_to_GIS utility.

Median

Use ASC_to_ASC utility (-statMedian switch) to create the median result of required output (water level, depth, velocity, hazard, ...) for each AEP and storm duration combination.

Maximums

Use ASC_to_ASC utility (-max switch) to create the maximum envelope of the median results for each AEP.

Processing Batch File Tips

  • The location of the batch file is recommended to be in the same folder as the result files. This will enable the use of relative path for the input and output files to make the command syntax shorter.
Example:
This method would be preferable:
"C:\TUFLOW\Utilities\asc_to_asc_w64.exe" -b -out ARR_Max.flt -max ARR_060m.flt ARR_120m.flt
Instead of this method:
"C:\TUFLOW\Utilities\asc_to_asc_w64.exe" -b -out "C:\Projects\Project_1024\Processing\ARR_Max.flt" -max "C:\Projects\Project_1024\Processing\ARR_060m.flt" "C:\Projects\Project_1024\Processing\ARR_120m.flt"
  • A wild card (*) can be used to avoid typing all the file names. Instead ASC_to_ASC utility will automatically read all the files with the same start and end of the filename.
Example:
This method would be preferable:
"C:\TUFLOW\Utilities\asc_to_asc_w64.exe" -b -out ARR_Max.flt -max ARR_*.flt
Instead of this method:
"C:\TUFLOW\Utilities\asc_to_asc_w64.exe" -b -out ARR_Max.flt -max ARR_060m.flt ARR_120m.flt ARR_180m.flt ARR_240m.flt ARR_300m.flt ARR_360m.flt
  • Looping through variables in a batch file can also be used to make the processing more efficient. Example of such batch file is below:
@echo off

:: This sets the variables as local, so you can use another batch file with A and B variables
SetLocal

:: set path to ASC_to_ASC Utility
set ASC_to_ASC="C:\TUFLOW\Utilities\asc_to_asc_w64.exe"

:: set up variables
set A=060m 120m 180m 240m 300m 360m
set B=tp01 tp02 tp03 tp04 tp05 tp06 tp07 tp08 tp09 tp10

:: Loop Through
FOR %%a in (%A%) do (
    FOR %%b in (%B%) DO (
		%ASC_to_ASC% -b -statMedian ARR_%%a_%%b_001.flt
    )
)
pause