TUFLOW 2D Cell Size Selection: Difference between revisions
Content deleted Content added
Chris Huxley (talk | contribs) No edit summary |
Chris Huxley (talk | contribs) No edit summary |
||
Line 167:
* Detailed integrated 1D/2D urban modelling may adopt the 15ft (4.6m) or slightly finer 12ft (3.7m) cell resolution if the cumulative simulation time for all model events/scenarios does not adversely impact the project timeline. This timeframe consideration is particularly relevant in locations such as Australia where an ensemble methodology is currently replacing single event approaches as a way to address uncertainty associated with hydrologic assumptions.
* Broad-scale quick run simulation for urban flood forecasting purposes may adopt a coarser 20ft cell resolution or investigate the suitability of a 25ft cell resolution to achieve a quicker simulation turnaround time.<br>
==How much will my runtime change when changing cell size?==
Solution time for a model is dependent on a number of factors: the solution scheme used, the size and duration of the model, the nature of the hydraulics within the model, the model features selected, and the hardware that it is executed on. However, once a model is developed at a relatively coarse resolution, there are some guidelines that can assist with predicting what the solve time will be as the resolution is refined. These can be useful for estimating whether the refined model will run overnight or need a few days.<br>
TUFLOW classic uses an implicit solution scheme that sweeps along columns and rows of cells. The solution scheme is one of the most efficient ever devised for solving the Shallow Water Equations (SWEs), but cannot be parallelised as the order of calculations are not independent. As a result the code is written to execute on a single core of a CPU. The time required to solve a model is almost exactly proportional to the number of wet cells in the model and the number of timesteps required. When the model cell size is halved, the number of cells increases by a factor of 4, and the timestep reduces by a factor of 2, so the model solve time increases by very close to a factor of 8.<br>
TUFLOW HPC uses an explicit solution scheme, for which the cell-based calculations are able to be performed in parallel. This can be done using multiple cores of the CPU, but it is more efficiently done using the many thousands of cores of modern GPUs. The model is cut into small tiles that are divided up amongst the streaming multi-controllers (SMs) of the GPU, with each SM distributing the cell calculations of its tile amongst its cores. However, there are still some calculations that must be done at the ‘top-level’ of the model, such as updating boundary flows and levels, extracting the plot output data, timestep control, and the process of configuring and launching the GPU parallel code. These tasks remain on the CPU and cannot be parallelised as the order of execution of these instructions is vitally important. So the overall model execution contains a mix of both parallel and serial work, with the total model solve time being the sum of both. As a result, the solve time for a model doesn’t follow a simple scaling rule. The cell-based component is still proportional to the number of cells times the number of timesteps, whereas the serial component of the work is generally only proportional to the number of timesteps. To be correct, this was also true of the single core solver for TUFLOW classic, but because the cell-based component was generally more than 99% of the work load, the scaling followed the simple rule of 8 times longer for half the cell size.<br>
In summary, for small models running with TUFLOW HPC, the solution time will increase by a factor somewhere in the range of 3 to 8 for a halving of the cell size. Once the model is sufficiently large that the cell-based calculations (running in parallel on the GPU) are taking most of the time, the time increase factor becomes close to 8. Exactly what is a “large” model will depend of the relative speed of the GPU and CPU. For that latest GPUS (as of 2021 this means the Nvidia A100 and RTX 30xx series) large typically means well over a million cells.<br>
However, there is one last and important caveat when refining cell size. The explicit algorithm used by TUFLOW HPC requires that the timestep is also subject to Peclet (diffusion) number control, not just Courant (velocity) number control. As Peclet number is inversely proportional to cell size squared, whereas Courant number is just inversely proportional to cell size, the model will transition from Courant number control to Peclet number control once the cell size becomes small enough. This means that the factor of 8 on time for a halving of cell size will eventually become a factor of 16. This was not a feature of TUFLOW Classic. In general, we do not recommend refining cell size well into the region where timestep control is completely dominated by the Peclet number (the Nd column in the HPC output). The flows in the primary channels are usually well converged by this point, and the 2D map outputs may not reveal a lot of additional detail in the velocity fields.<br>
<br>
<br>
<br>
| |||