Developing Dynamic Load Balancing library for wsiFoam

eCSE12-08

Key Personnel

PI/Co-I: Prof. Deborah Greaves (University of Plymouth)

Technical: Dr. Xiaohu Guo (STFC), Dr. Scott Brown (University of Plymouth)

Relevant documents

eCSE Technical Report: Developing Dynamic Load Balancing library for wsiFoam

Project summary

Offshore and coastal engineering fields are using increasingly larger and more complex numerical simulations to model wave-structure interaction (WSI) problems, in order to improve understanding of safety and cost implications. Therefore, an efficient multi-region WSI toolbox, wsiFoam, is being developed within an open-source community-serving numerical wave tank (NWT) facility based on the computational fluid dynamics (CFD) code OpenFOAM®, as part of the Collaborative Computational Project in Wave Structure Interaction (CCP-WSI).

However, even using the efficiency of a multi-region approach, the computational expense of CFD is high, and hence there is a constant need to improve the efficiency of these high-fidelity codes. One way that this can be achieved is through the parallelisation of code to make use of high performance computing facilities. Although OpenFOAM® is parallel already, historically the MPI performance has been considered sub-optimal, but recent developments have led to a number of performance improvements being implemented in OpenFOAM® v5.x along with new parallel I/O functionality.

The developments have led to a significant performance benefit when employing a large number of processors, and it is vital that existing code is updated to be compatible with OpenFOAM® v5.x in order to utilise this functionality. However, OpenFOAM® v5.x still only offers static domain decomposition, and this limits the choice of parallel load balancing methods to algorithms that can only take basic user defined arguments to aid in load balancing. These methods typically use blocking MPI communications that only consider the number of mesh cells and their spatial distribution (ie they assume homogeneity of load per mesh cell). As typical WSI simulations are often inhomogeneous with respect to the mesh, due to properties such as mesh motion and wave 'relaxation', these decomposition methods based purely on mesh resolution are likely to be sub-optimal in terms of computing resource usage.

Therefore in this work the wsiFoam toolbox was updated to utilise the new parallel I/O functionality in OpenFOAM® v5.x. Furthermore, it is coupled with a newly implemented dynamic load balancing method, based on a new ParMETIS decomposition class, which considers all of the relevant parameters required for efficient simulation and optimal parallel performance of WSI problems. Benchmarking of the new functionality shows that the load imbalances are a major performance bottleneck for dynamic mesh applications. Comparison with the existing 'dam break with obstacle' test case (see Figure 1) shows a nearly five-times speedup using 1536 cores with 17 million mesh cells on ARCHER.

Achievement of objectives

The objectives of the work were to:

1. Successfully ported wsiFoam to OpenFOAMŪ v5.x to take advantage of recently implemented improved parallel input/output (I/O).
2. Implemented a new domain decomposition class parmetisDecomp, which uses the ParMETIS algorithm to determine how a simulation should be distributed. ParMETIS extends the functionality provided by METIS and includes routines that are especially suited for parallel AMR computations and large-scale numerical simulations. The main purpose of this objective is to reduce partition overhead for dynamic load balancing.
3. Implemented a new dynamic load balancing library, dynamicBalancedFvMesh, which is compatible with the wsiFoam toolbox. The implementation is based on the dynamicFvMesh class but adds mesh balancing for parallel cases to the mesh.update() function. This library has now enabled wsiFoam to scale to more than a thousand cores and can be also used by other dynamic mesh OpenFOAM applications.

Summary of the Software

The eCSE-1208 project software packages use git to manage software revision control. There are two major software packages involved in this project: wsiFoam code and OpenFOAM-5.x.

The newly ported wsiFoam software (compatible with OpenFOAM® v5.x) is available at the CCP-WSI repository: https://www.ccp-wsi.ac.uk/code_repository

The new domain decomposition class which uses ParMETIS is available on github using a fork of the official OpenFOAM v5.x repository:

https://github.com/guoxiaohu/OpenFOAM-5.x

https://github.com/guoxiaohu/ThirdParty-5.x

The dynamic load balancing library development repository is currently held at bitbucket (https://bitbucket.org/guoxh/dynamicbalancedfvmesh/src/master/). Once granted access permission, ARCHER users can easily use git to check out the master branch. The plan is to release it together with the wsiFoam toolbox on the CCP-WSI repository.

Scientific Benefits

Linear or second-order wave diffraction theory is the classical approach for analysing wave interactions with fixed or floating structures. However this approach cannot represent the highly nonlinear effects associated with extreme sea states including the violent motion and break-up of the water-air interface and the hydro-elastic coupling between structures and waves.

As an improved understanding of these interactions is essential for engineers to design safe offshore structures, these problems are often evaluated through laboratory experiments and sea trials, which can be time consuming and costly. Furthermore, fully nonlinear simulations based only on potential flow theory do not include viscosity, turbulence and air-water compressible/incompressible interactions.

Capturing the physical processes involved in extreme wave interaction requires a detailed CFD simulation of the full Navier-Stokes (NS) equations. The flow model should include both fluids, compressible air and incompressible water as required, aeration as waves break and impact a structure, floating bodies, wind effects on waves, turbulence and the fully coupled hydro-elastic response of a compliant structure. These computations are extremely compute-intensive, currently prohibiting their use in routine design processes.

As reinforced by the CCP-WSI Focus Group Workshop (14th July 2016, City University), larger and more complex CFD simulations are required by the WSI industry and academic community. The Workshop discussed the state of the art, current capabilities and future challenges in WSI and produced a priority list of challenges for CFD development (as reported in https://www.ccp-wsi.ac.uk/sites/www.ccp-wsi.ac.uk/files/CCP-WSIFocusGroupWS1_Report.pdf). Unsurprisingly, the highest priority was given to the challenge of developing methods for complex and more efficient computations. In tackling this, the CCP-WSI members are developing the NWT following a zonal CFD approach as explained above. A key aspect is the interface between different zones and efficient implementation of the interfaced solvers on an HPC platform.

The profiling and parallel optimisation work conducted in this project provides the WSI community with a better understanding of the parallel behaviour and scalability of zonal CFD WSI simulations. These activities have identified the critical factors influencing code performance and ultimately lead to improvements in I/O routines and data storage/reading. This will reduce the limitations presently restricting the design of WSI experiments and enable new software developments to be made in an area of high national importance.

These improvements will aid in the routine application of these new methods in the early design of offshore structures, ultimately providing significant cost savings to the industry, promoting their use on a day-to-day basis and increasing the demand for relevant technical expertise. The new load balancing techniques developed and the associated parallel implementations will also benefit the general CFD community as a whole (other researchers in academia or commercial organisations working in any area involving free surface flows, more generally within CFD and related areas of computational physics) by providing recommendations to relieve bottlenecks and improve the load distribution of similar applications.

Although focused on wsiFoam, this work benefits the entire OpenFOAM® community enabling more sophisticated, open-source numerical modelling to be undertaken by exploiting next generation HPC resources. The work conducted in this eCSE project helps to identify and relieve some of the barriers restricting the large-scale implementation of OpenFOAM®. This alone will improve the scalability of many OpenFOAM® applications and enable new science to be performed in a wide variety of disciplines.

Figure 1: Snapshots of the volume of fluid iso-surface (α1 = 0.5) from the dam break with obstacle case using eight processors, adaptive mesh and dynamic load balancing. The colours represent the sections being solved by each processor, and the grey lines show the mesh discretisation.