HPC FAQ - Revision history

Pavlina Monhartova: /* Why is my HPC model getting unstable? Why am I getting a timestep error? Why are my control numbers so low? */

2024-11-20T01:14:12Z

Why is my HPC model getting unstable? Why am I getting a timestep error? Why are my control numbers so low?

← Older revision		Revision as of 11:14, 20 November 2024
Line 100:		Line 100:
	The below suggestions can be implemented to eliminate the instability and/or the decrease in control numbers:		The below suggestions can be implemented to eliminate the instability and/or the decrease in control numbers:
	* Specify initial water level for the whole model with <font color="blue"><tt>Set IWL</tt></font> command or locally with <font color="blue"><tt>Read GIS IWL</tt></font> command. The wet cells can limit the adaptive timestep through the <u>[[HPC_Adaptive_Timestepping#HPC_2D_Timestep \| Shallow Wave Celerity Number]]</u>, and prevent the HPC solver from using big timesteps.		* Specify initial water level for the whole model with <font color="blue"><tt>Set IWL</tt></font> command or locally with <font color="blue"><tt>Read GIS IWL</tt></font> command. The wet cells can limit the adaptive timestep through the <u>[[HPC_Adaptive_Timestepping#HPC_2D_Timestep \| Shallow Wave Celerity Number]]</u>, and prevent the HPC solver from using big timesteps.
	* Use <font color="blue"><tt>Timestep Maximum</tt></font> command to cap the maximum timestep to not get too high. A good Timestep Maximum value to start with might be a half the cell size in metres, e.g. if the cell size is 5m, the Timestep Maximum is to be 2.5 seconds. The .hpc.tlf file can be checked of further refinement is needed.		* Use <font color="blue"><tt>Timestep Maximum</tt></font> command to cap the maximum timestep to not get too high. A good Timestep Maximum value to start with might be a half the cell size in metres, e.g. if the cell size is 5m, the Timestep Maximum is to be 2.5 seconds. The .hpc.tlf file can be checked if further refinement is needed.
	<br>		<br>

Pavlina Monhartova: /* Why is my HPC model getting unstable? Why am I getting a timestep error? Why are my control numbers so low? */

2024-11-20T01:13:39Z

Why is my HPC model getting unstable? Why am I getting a timestep error? Why are my control numbers so low?

← Older revision		Revision as of 11:13, 20 November 2024
Line 100:		Line 100:
	The below suggestions can be implemented to eliminate the instability and/or the decrease in control numbers:		The below suggestions can be implemented to eliminate the instability and/or the decrease in control numbers:
	* Specify initial water level for the whole model with <font color="blue"><tt>Set IWL</tt></font> command or locally with <font color="blue"><tt>Read GIS IWL</tt></font> command. The wet cells can limit the adaptive timestep through the <u>[[HPC_Adaptive_Timestepping#HPC_2D_Timestep \| Shallow Wave Celerity Number]]</u>, and prevent the HPC solver from using big timesteps.		* Specify initial water level for the whole model with <font color="blue"><tt>Set IWL</tt></font> command or locally with <font color="blue"><tt>Read GIS IWL</tt></font> command. The wet cells can limit the adaptive timestep through the <u>[[HPC_Adaptive_Timestepping#HPC_2D_Timestep \| Shallow Wave Celerity Number]]</u>, and prevent the HPC solver from using big timesteps.
	* Use <font color="blue"><tt>Timestep Maximum</tt></font> command to cap the maximum timestep to not get too high.		* Use <font color="blue"><tt>Timestep Maximum</tt></font> command to cap the maximum timestep to not get too high. A good Timestep Maximum value to start with might be a half the cell size in metres, e.g. if the cell size is 5m, the Timestep Maximum is to be 2.5 seconds. The .hpc.tlf file can be checked of further refinement is needed.
	<br>		<br>

Emilie Nielsen at 00:36, 23 September 2024

2024-09-23T00:36:18Z

← Older revision		Revision as of 10:36, 23 September 2024
Line 84:		Line 84:
	= With Wu turbulence scheme being the new default, are old models using Smagorinsky wrong? =		= With Wu turbulence scheme being the new default, are old models using Smagorinsky wrong? =
	Turbulence is pronounced in areas of highly transient flow, e.g. high velocities, bends, ledges and flow contraction/expansion. Where the flow is more benign and/or bed roughness is high, turbulence is not so important as it only applies where there are strong spatial velocity gradients, for example, for uniform flow in a straight rectangular channel the turbulence term is zero as there is no spatial velocity gradient.<br>		Turbulence is pronounced in areas of highly transient flow, e.g. high velocities, bends, ledges and flow contraction/expansion. Where the flow is more benign and/or bed roughness is high, turbulence is not so important as it only applies where there are strong spatial velocity gradients, for example, for uniform flow in a straight rectangular channel the turbulence term is zero as there is no spatial velocity gradient.<br>
	The problem with Smagorinsky, which is a large scale eddy turbulence model originally developed for coastal modelling, is that it is cell size dependent (is proportional to cell surface area) and tends to zero as the cell size tends to zero. This has historically not been a major issue as cell sizes have typically been greater than the depth. The general recommendation in the TUFLOW Manual is to be careful of using cell sizes significantly smaller than the depth (see Section 1.4). However, as cells have been becoming finer and finer with the advent of GPU models this issue has increasingly emerged, and is particularly pertinent if using a quadtree or flexible mesh and very small cells relative to the depth are being used. If this is the case, bigger differences will be present for bigger events where the water level and velocities are higher.<br>		The problem with Smagorinsky, which is a large scale eddy turbulence model originally developed for coastal modelling, is that it is cell size dependent (is proportional to cell surface area) and tends to zero as the cell size tends to zero. This has historically not been a major issue as cell sizes have typically been greater than the depth. The general recommendation in the <u>[https://docs.tuflow.com/classic-hpc/manual/latest/ TUFLOW Manual]</u> is to be careful of using cell sizes significantly smaller than the depth (see Section 1.4). However, as cells have been becoming finer and finer with the advent of GPU models this issue has increasingly emerged, and is particularly pertinent if using a quadtree or flexible mesh and very small cells relative to the depth are being used. If this is the case, bigger differences will be present for bigger events where the water level and velocities are higher.<br>
	TUFLOW, many years ago, changed from purely Constant or purely Smagorinsky to Smagorinsky plus (a small amount of) Constant. This improved absorption of eddies into the streamlines behind a bluff body and helped by varying degrees the modelling at finer cell sizes. This was working well in the time being, however new cell size turbulence scheme has now been implemented to help with the situation even further. <br>		TUFLOW, many years ago, changed from purely Constant or purely Smagorinsky to Smagorinsky plus (a small amount of) Constant. This improved absorption of eddies into the streamlines behind a bluff body and helped by varying degrees the modelling at finer cell sizes. This was working well in the time being, however new cell size turbulence scheme has now been implemented to help with the situation even further. <br>
	The Smagorinsky/Constant turbulence combination has served the industry well and can continue to be used where the cell sizes are not significantly smaller than the depth where highly transient flows are occurring. If the model is well calibrated (using conventional parameters), continuing to use the Smagorinsky/Constant turbulence option is fine. Therefore, it is not considered that TUFLOW (or other good 2D solvers) have been producing questionable results, but that an improved turbulence representation is needed for 2D schemes with fine-scale cells. This is especially the case for the new Quadtree mesh option and for flexible meshes that utilise fine-scale cells.<br>		The Smagorinsky/Constant turbulence combination has served the industry well and can continue to be used where the cell sizes are not significantly smaller than the depth where highly transient flows are occurring. If the model is well calibrated (using conventional parameters), continuing to use the Smagorinsky/Constant turbulence option is fine. Therefore, it is not considered that TUFLOW (or other good 2D solvers) have been producing questionable results, but that an improved turbulence representation is needed for 2D schemes with fine-scale cells. This is especially the case for the new Quadtree mesh option and for flexible meshes that utilise fine-scale cells.<br>

Chris Huxley: /* Should I see 100% GPU utilisation when no other processes are running on GPU? */

2024-08-05T23:13:00Z

Should I see 100% GPU utilisation when no other processes are running on GPU?

← Older revision		Revision as of 09:13, 6 August 2024
Line 114:		Line 114:

	= Should I see 100% GPU utilisation when no other processes are running on GPU? =		= Should I see 100% GPU utilisation when no other processes are running on GPU? =
	HPC still does a small amount of work on CPU such as the model initiation and the final step of data reduction for model volume, control numbers, and stability checks. Frequent map outputs specifically for large datasets might also contribute to lower GPU utilisation as writing of outputs happens on CPU. Even in a perfect world and 2D only model it isn't possible to see 100% GPU utilisation. If there are any 1D features in the model the GPU utilisation will be even lower as 1D is processed on CPU only. A model with 1D ESTRY connection can potentially be doing a lot of work on CPU, perhaps as much as 90% CPU and 10% GPU. If the CPU hardware is not matched correctly with the GPU card it can become a bottleneck for HPC GPU runs even with a few 1D elements~~. We are investigating the possibility of parallelising 1D for future releases so it is able to run on GPU~~.<br>		HPC still does a small amount of work on CPU such as the model initiation and the final step of data reduction for model volume, control numbers, and stability checks. Frequent map outputs specifically for large datasets might also contribute to lower GPU utilisation as writing of outputs happens on CPU. Even in a perfect world and 2D only model it isn't possible to see 100% GPU utilisation. If there are any 1D features in the model the GPU utilisation will be even lower as 1D is processed on CPU only. A model with 1D ESTRY connection can potentially be doing a lot of work on CPU, perhaps as much as 90% CPU and 10% GPU. If the CPU hardware is not matched correctly with the GPU card it can become a bottleneck for HPC GPU runs even with a few 1D elements.<br>
	<br>		<br>

ElizaCollison: /* Why does my TUFLOW HPC simulation take longer than TUFLOW Classic? */

2024-05-24T12:48:18Z

Why does my TUFLOW HPC simulation take longer than TUFLOW Classic?

← Older revision		Revision as of 22:48, 24 May 2024
Line 28:		Line 28:
	'''Over utilisation of CPU threads/cores'''<br>		'''Over utilisation of CPU threads/cores'''<br>
	Trying to run multiple HPC simulations across the same CPU threads. If, for example, you have 4 CPU threads on your computer and you run two simulations that both request 4 threads, then effectively you are overloading the CPU hardware by requesting 8 threads in total. This will slow down the simulations by more than a factor of 2. The most efficient approach in this case is to run both simulations using 2 threads each, noting that if you are performing other CPU intensive tasks, this also needs to be considered.<br>		Trying to run multiple HPC simulations across the same CPU threads. If, for example, you have 4 CPU threads on your computer and you run two simulations that both request 4 threads, then effectively you are overloading the CPU hardware by requesting 8 threads in total. This will slow down the simulations by more than a factor of 2. The most efficient approach in this case is to run both simulations using 2 threads each, noting that if you are performing other CPU intensive tasks, this also needs to be considered.<br>
	By default, the number of CPU threads taken is two (2). You can control the number of threads requested by either using the <font color="blue"><tt>-nt<number_threads></tt></font> run time option, e.g. -nt2, or use the TCF command <font color="blue"><tt>CPU Threads</tt></font>. The -nt run time option overrides <font color="blue"><tt>CPU Threads</tt></font>.<br>		By default, from the 2020-01 release onwards the number of CPU threads taken is four (4). Previously, the default was two (2). You can control the number of threads requested by either using the <font color="blue"><tt>-nt<number_threads></tt></font> run time option, e.g. -nt2, or use the TCF command <font color="blue"><tt>CPU Threads</tt></font>. The -nt run time option overrides <font color="blue"><tt>CPU Threads</tt></font>.<br>
	Note: If Windows hyperthreading is active there typically will be two threads for each physical core. For computationally intensive processes such as TUFLOW, it is recommended that hyperthreading is deactivated so there is one thread for each core.<br>		Note: If Windows hyperthreading is active there typically will be two threads for each physical core. For computationally intensive processes such as TUFLOW, it is recommended that hyperthreading is deactivated so there is one thread for each core.<br>

Pavlina Monhartova: /* Why is my HPC model getting unstable? And why are my control numbers lower than they could be? */

2024-05-21T22:24:05Z

Why is my HPC model getting unstable? And why are my control numbers lower than they could be?

← Older revision		Revision as of 08:24, 22 May 2024
Line 90:		Line 90:
	<br>		<br>

	= Why is my HPC model getting unstable? ~~And~~ ~~why~~ are my control numbers ~~lower than they could~~ be? =		= Why is my HPC model getting unstable? Why am I getting a timestep error? Why are my control numbers so low? =
	If there are no wet cells at the beginning of the simulation, the adaptive timestep can get quite big. Once the flow increases rapidly, instabilities can develop, which leads to oscillations in variables that grow over time, eventually leading to NaNs in the solution. Two situations can occur:		If there are no wet cells at the beginning of the simulation, the adaptive timestep can get quite big. Once the flow increases rapidly, instabilities can develop, which leads to oscillations in variables that grow over time, eventually leading to NaNs in the solution. Two situations can occur:
	* The solver can't reduce the big timestep to a sufficiently small timestep within the ten default trials, the simulation gets unstable and the model is stopped with an error.		* The solver can't reduce the big timestep to a sufficiently small timestep within the ten default trials, the simulation gets unstable and the model is stopped with an error.

Pavlina Monhartova: /* Is it possible run two TUFLOW HPC simulations using a single GPU and how does this affect performance? */

2023-12-15T01:58:29Z

Is it possible run two TUFLOW HPC simulations using a single GPU and how does this affect performance?

← Older revision		Revision as of 11:58, 15 December 2023
Line 79:		Line 79:

	= Is it possible run two TUFLOW HPC simulations using a single GPU and how does this affect performance? =		= Is it possible run two TUFLOW HPC simulations using a single GPU and how does this affect performance? =
	Yes, it is possible to run two (and more) than one simulation on a single GPU. The performance depends on whether the model needs more than half of the GPU’s resources. At worst, the models complete just as fast as if they ran one after the other. At best, the GPU has enough resources to run both side by side and complete them as if they ran on two GPUs. Realistically, the model may use a bit more than half of the resources, causing some slowdown. Model run’s startup phase can require 4-6x the RAM than the model calculations, the available memory needs to be sufficient to handle two (or more) models starting up side by side. If it is not sufficient, the model's startup phases can be staggered.		Yes, it is possible to run two (and more) than one simulation on a single GPU. The performance depends on whether the model needs more than half of the GPU’s resources. At worst, the models complete just as fast as if they ran one after the other. At best, the GPU has enough resources to run both side by side and complete them as if they ran on two GPUs. Realistically, the model may use a bit more than half of the resources, causing some slowdown. Model run’s startup phase can require 4-6x the RAM than the model calculations, the available memory needs to be sufficient to handle two (or more) models starting up side by side. If it is not sufficient, the model's startup phases can be staggered.<br>
			<br>

	= With Wu turbulence scheme being the new default, are old models using Smagorinsky wrong? =		= With Wu turbulence scheme being the new default, are old models using Smagorinsky wrong? =

Pavlina Monhartova: /* Why is my model using post 2020 HPC slower than pre 2020 HPC? */

2023-12-15T01:58:02Z

Why is my model using post 2020 HPC slower than pre 2020 HPC?

← Older revision		Revision as of 11:58, 15 December 2023
Line 77:		Line 77:
	Despite the possible increase in runtime for some models, the Wu turbulence scheme is warranted particularly as cell sizes are typically getting smaller.<br>		Despite the possible increase in runtime for some models, the Wu turbulence scheme is warranted particularly as cell sizes are typically getting smaller.<br>
	<br>		<br>

			= Is it possible run two TUFLOW HPC simulations using a single GPU and how does this affect performance? =
			Yes, it is possible to run two (and more) than one simulation on a single GPU. The performance depends on whether the model needs more than half of the GPU’s resources. At worst, the models complete just as fast as if they ran one after the other. At best, the GPU has enough resources to run both side by side and complete them as if they ran on two GPUs. Realistically, the model may use a bit more than half of the resources, causing some slowdown. Model run’s startup phase can require 4-6x the RAM than the model calculations, the available memory needs to be sufficient to handle two (or more) models starting up side by side. If it is not sufficient, the model's startup phases can be staggered.

	= With Wu turbulence scheme being the new default, are old models using Smagorinsky wrong? =		= With Wu turbulence scheme being the new default, are old models using Smagorinsky wrong? =

ElizaCollison at 05:08, 13 February 2023

2023-02-13T05:08:30Z

@@ Line 119: / Line 119: @@
 *An explicit scheme that is parallelised will run a single simulation faster by around a factor of 5 on an 8 core machine – you will never get a mark-up of 8 on an 8 core machine as there is an overhead in managing the computations across the cores.
 *Users are often doing two or more simulations at the same time.  For example different events (100, 20 year…, different durations, etc).  In these situations, even if a scheme is parallelised, it is usually better and sometimes much better, to run each simulation unparallelised on their own core.  For example, if you have four simulations and four cores, definitely don’t run them parallelised, but run all four at once unparallelised.  If a fifth simulation is started up this will then slow down the other simulations.<br>
 <br>

Pavlina Monhartova: /* Should I see 100% GPU utilisation when no other processes are running on GPU? */

2022-07-14T02:15:12Z

Should I see 100% GPU utilisation when no other processes are running on GPU?

← Older revision		Revision as of 12:15, 14 July 2022
Line 110:		Line 110:

	= Should I see 100% GPU utilisation when no other processes are running on GPU? =		= Should I see 100% GPU utilisation when no other processes are running on GPU? =
	HPC still does a small amount of work on CPU such as the model initiation and the final step of data reduction for model volume, control numbers, and stability checks. Frequent map outputs specifically for large datasets might also contribute to lower GPU utilisation as writing of outputs happens on CPU. Even in a perfect world and 2D only model it isn't possible to see 100% GPU utilisation. If there are any 1D features in the model the GPU utilisation will be even lower as 1D is processed on CPU only. A model with 1D ESTRY connection can potentially be doing a lot of work on CPU, perhaps as much as 90% CPU and 10% GPU. If the CPU hardware is not matched correctly with the GPU card it can become a bottleneck for HPC-GPU runs even with a few 1D elements. We are investigating the possibility of parallelising 1D for future releases so it is able to run on GPU.<br>		HPC still does a small amount of work on CPU such as the model initiation and the final step of data reduction for model volume, control numbers, and stability checks. Frequent map outputs specifically for large datasets might also contribute to lower GPU utilisation as writing of outputs happens on CPU. Even in a perfect world and 2D only model it isn't possible to see 100% GPU utilisation. If there are any 1D features in the model the GPU utilisation will be even lower as 1D is processed on CPU only. A model with 1D ESTRY connection can potentially be doing a lot of work on CPU, perhaps as much as 90% CPU and 10% GPU. If the CPU hardware is not matched correctly with the GPU card it can become a bottleneck for HPC GPU runs even with a few 1D elements. We are investigating the possibility of parallelising 1D for future releases so it is able to run on GPU.<br>
	<br>		<br>