View source for HPC Running and Converting Models

= Introduction =
This page contains information about converting an existing TUFLOW Classic or GPU (pre 2017 HPC release) model to a format that can be run using the TUFLOW HPC engine. This page provides a quick summary for experienced TUFLOW users to use as a reference point for updating their models. It is recommended that less experienced TUFLOW users refer to our <u>[[Tutorial_Model |TUFLOW Tutorial Modules]]</u> for greater support and guidance on creating a HPC model.<br>

To convert an existing TUFLOW Classic or GPU Model to run on HPC, an update to the TUFLOW Control File (TCF) is needed. Some features from TUFLOW Classic that are not currently supported in HPC, may prevent the HPC model running successfully. To find out more about unsupported features in HPC be sure to review the <u>[https://tuflow.com/Download/TUFLOW/Releases/2017-09/TUFLOW%20Release%20Notes.2017-09.pdf TUFLOW 2017-09 Release Notes]</u> or the <u>[[HPC Features | HPC Features Wiki Page]]</u>.<br>

= Converting TUFLOW Classic to HPC (TCF Updates) =
To run an existing TUFLOW Classic simulation with the new HPC engine, the following lines of text need to be added to the TUFLOW Control File (TCF).<br>
<font color="blue"><tt>Solution Scheme </tt></font> <font color="red"><tt>==</tt></font><tt> HPC </tt> <font color="green"><tt>   !This command specifies that you want to run TUFLOW using the HPC solution scheme or engine.</tt></font><br>
The following command is also required to run the model using GPU hardware:<br> 
<font color="blue"><tt>Hardware </tt></font> <font color="red"><tt>==</tt></font><tt> GPU </tt>        <font color="green"><tt>   !CPU is default. The hardware command instructs TUFLOW HPC to run using GPU hardware. This is typically orders of magnitude faster than on CPU.</tt></font><br>
These two commands are all that's needed to run convert the TUFLOW Classic model to HPC and run using GPU hardware. There are however more commands provided below that give the modeller greater control over the Hardware that HPC uses.<br>

= Running HPC on Multiple CPU Threads =
As mentioned in the <u>[[HPC_Introduction | HPC Introduction]]</u> page, HPC can be parallelised to run across multiple CPU processors when run on CPU (i.e. not GPU). The following command allows the modeller to dictate the number of core processors to run TUFLOW HPC across.<br>
<font color="blue"><tt>CPU Threads </tt></font> <font color="red"><tt>==</tt></font><tt> 8 </tt>  <font color="green"><tt>   !Default is 4. This instructs TUFLOW to search for and run the model across four different core processors. </tt></font><br>
If the number of processors or TUFLOW licences found by TUFLOW are less than the specified value, then TUFLOW will utilise the maximum number of core processors available within the licence and hardware limitations.<br>
Alternatively, the number of CPU threads can be specified in the batch file / command line by using the -nt<number of threads> argument. If both control file and command line are used to specify number of threads, the command line option will prevail.

= Running HPC on Multiple GPU Devices =
Much like HPC can be run across multiple CPU processors, HPC can be run across multiple GPU cards. Models can also be instructed to run on a specific GPU card.<br>
If a machine only has a single GPU card, the GPU Device ID should be 0, this is a default. If a second GPU card was added, the Device ID would be 1 and so on. The GPU IDs can be checked by reviewing the machines ''Device Manager''.<br>
The most common is to specify the GPU card ID in the batch file / command line by using the -pu<id> argument.<br>
Example below will run single simulation across the first and second GPU card (ID 0 and 1). 
<pre>"TUFLOW_iSP_w64.exe" -pu0 -pu1 "M01_5m_001.tcf"</pre>
Example below will run single simulation on the fourth GPU card (ID 3). 
<pre>"TUFLOW_iSP_w64.exe" -pu3 "M01_5m_001.tcf"</pre>

Alternatively, the following TCF command can be used to set the number of GPU devices and which devices to use.<br>
<font color="blue"><tt>GPU Device IDs </tt></font> <font color="red"><tt>==</tt></font><tt> 0, 1 </tt>	<font color="green"><tt>   !This command instructs TUFLOW to run the model on GPU Device 0 and GPU Device 1.</tt></font><br>
If both control file and command line are used to specify devices, the command line option will prevail.

= Converting TUFLOW GPU to HPC (TCF Updates) =
When converting a TUFLOW GPU model across to HPC, first confirm that all features in the GPU model are available in HPC by referring to the <u>[https://tuflow.com/Download/TUFLOW/Releases/2017-09/TUFLOW%20Release%20Notes.2017-09.pdf TUFLOW 2017-09 Release Notes]</u>.<br>
Delete the following command from the *.tcf file and insert the commands specified above, for converting a TUFLOW Classic model to HPC.<br>
<s><font color="blue"><tt>GPU Solver </tt></font> <font color="red"><tt>==</tt></font><tt> ON </tt></s><br>

= HPC Scenarios and Updating the Batch File =
Modellers may want to change the hardware that HPC is run on throughout the course of a project. For example, if your company own more CPU than GPU licences it may be beneficial to run the model using CPU hardware during the initial model build phase so your colleagues have access to the higher speed GPU licences for production runs on other projects that are running in parallel. If this is the case, it may be easier to setup a Scenario Logic statement in the TUFLOW Control File (TCF), that allows the modeller to change the hardware being used with a simple switch in the batch file used to run the model.

To setup a scenario for varying hardware options, the following commands can be used in the TCF file:
<font color="blue"><tt>Solution Scheme </tt></font> <font color="red"><tt>==</tt></font><tt> HPC </tt> <br>
<font color="blue"><tt>Hardware </tt></font> <font color="red"><tt>==</tt></font> <tt> <<~s1~>> </tt>    <font color="green"><tt>   !The scenario will either be "CPU" or "GPU"</tt></font><br>

This basic scenario logic can be configured further within the TCF as shown below:<br>

<font color="blue"><tt>If Scenario </tt></font> <font color="red"><tt>==</tt></font> <tt> CPU </tt> <br>
:: <font color="blue"><tt>CPU Threads </tt></font> <font color="red"><tt>==</tt></font> <tt> 8 </tt> <br>
<font color="blue"><tt>Else If Scenario </tt></font> <font color="red"><tt>==</tt></font> <tt> GPU </tt> <br>
:: <font color="blue"><tt>GPU Device IDs</tt></font> <font color="red"><tt>==</tt></font> <tt> 0, 1 </tt> <br>
<font color="blue"><tt>End If </tt></font> <br>

If using the above Scenario Logic, the modeller must include a scenario call in the batch file.<br>
<font color="blue"><tt>-s1 <Hardware Type> </tt></font><br>
If you are unfamiliar with using Scenario Logic, please refer to <u>[[Tutorial_Module05 |Tutorial Module 05]]</u>.<br>

Other useful batch file switches include:<br>
<font color="blue"><tt>-nt <number_of_threads> </tt></font>: This switch is used to set the number of CPU threads used for CPU mode simulations.<br>
<font color="blue"><tt>-pu <GPU Device IDs> </tt></font>: This switch is used to select the number of GPU cards that TUFLOW uses.<br>
<br>
<br>
{{Tips Navigation
|uplink=[[ HPC_Modelling_Guidance | Back to HPC Modelling Guidance]]
}}
An example of how this would be implemented into a simple batch file for CPU and GPU are shown below.<br>

'''<u>CPU</u>'''<br>
<tt>TUFLOW_iSP_w64.exe -s1 <font color="blue">CPU</font> <font color="red">-nt8</font> FMA_T2_~s1~_001.tcf</tt><br>
This example will run TUFLOW HPC on CPU using 8 CPU threads.<br>

'''<u>GPU</u>'''<br>
<tt>TUFLOW_iSP_w64.exe -s1 <font color="blue">GPU</font> <font color="red">-pu0 -pu1</font> FMA_T2_~s1~_001.tcf</tt><br>
This example will run TUFLOW HPC on GPU using 2 GPU cards.<br>
<br>