TPLS: 3D Decomposition and Gas/Liquid Flows

Key Personnel

PI/Co-I: Dr. Prashant Valluri - University of Edinburgh and Dr. Lennon Ó Náraigh - University College Dublin

Technical: Dr. David Scott, EPCC

Relevant Documents

eCSE Technical Report: TPLS: 3D Decomposition and Gas/Liquid Flows

Project summary

TPLS is a freely available CFD code which solves the Navier-Stokes equations for an incompressible two-phase flow. The level set method enables sharp changes in interfacial topology and is therefore ideally suited to gaining an understanding of problems of interest to the oil and gas industry as well as looking at the implications of the inherent instability of the interface in a two-phase system.

This eCSE project aimed to improve the architecture and performance of the software, reduce overall runtime, and improve usability and maintainability, opening up the software to a large user community. The main aims of this project were to introduce TPLS 3.0 with: i) Implementation of 3D domain decomposition, ii) Replacing all SOR/Jacobi hand-coded routines with PETSc and iii) Implementation of density-contrast flows and implementation of a choice of boundary conditions.

TPLS 3.0 has now been released as opensource. The addition of the PETSc routines for the momentum solvers and implementation of 3D decomposition have made the solver more efficient, 2 times faster than TPLS 2.0. TPLS 3.0 was built upon TPLS 2.0 and this ensured that our data and storage efficiencies (we already achieved with TPLS 2.0) are further enhanced.

Finally this work has resulted in a significant re-factoring of the TPLS software, improving maintainability by increasing the use of error checking, debugging options and the use of modules, functions and subroutines to minimise code duplication. This should enable future development work to be significantly quicker and easier to debug, as well as providing more runtime options, moving away from extensive use of hard-coded compilation options. As a result we are now able to provide TPLS 3.0 as a package available on ARCHER for all users.

Achievement of objectives

The project had four main objectives:

1. Objective 1: Full 3D domain decomposition, leading to improved strong scaling (80% parallel efficiency on 64 ARCHER nodes for a 5.6 million element problem)

   A full 3D domain decomposition has been implemented. The parallel efficiency on 64 nodes is a disappointing 20% but the speed-up relative to the 2D code is greater than 2.

2. Objective 2: Replace all SOR solvers with PETSc (50% speedup in each compute routine on 64 nodes)

   All the three momentum solvers, which were SOR-based, have been implemented with PETSc. However, the interface detection solvers have not. The reasons for this are set out in the Project Summary (section 3). The measurements were actually carried out on 32 nodes using the density contrast code on an 11 million grid point domain. The PETSc (Krylov) momentum solvers ran at only 2/3 the speed of the SOR solvers. The PETSc pressure solver ran at the same speed as the corresponding SOR solver. The expected speed-ups were based on the speed-up produced by the introduction of a PETSc solver into the pressure calculation during a previous project. Unfortunately, the implementation of the previous solver was faulty and correcting the error led to a substantial decrease in speed.

3. Objective 3: Implementation of density-contrast flows (demonstration gas/liquid flow calculation using 3D PETSc code)

   Density contrast flows have been implemented. A gas/liquid flow calculation has also been demonstrated using this TPLS 3.0 solver. A comparison with TPLS 2.0 has not been carried out because the result would be uninformative. It was noticed that TPLS 2.0 had incorporated a faulty pressure solver which has a significant impact on the execution speed of the code. This has been corrected in TPLS 3.0.

   For the 11 million point problem running on 32 nodes TPLS 3.0 was 18% faster than TPLS 1.0 when using the SOR solvers. The PETSc solvers were 39% slower. The implementation has been validated for the case of the Rayleigh-Taylor instability.

4. Objective 4: Release TPLS 3.0

   TPLS 3.0 has been released via Sourceforge.

Summary of the software

TPLS is a powerful and efficient 3D Direct Numerical Simulation (DNS) flow solver to simulate multiphase flows at unprecedented detail, speed and accuracy.

This flow solver has been developed by Lennon Ó Náraigh (School of Mathematical Sciences, University College Dublin), Prashant Valluri (School of Engineering, University of Edinburgh), David Scott, Toni Collis and Iain Bethune (the Edinburgh Parallel Computing Centre at the University of Edinburgh) and Peter Spelt (Université de Lyon1, Claude Bernard) under the aegis of several HECToR computer time grants and funding from EPSRC, the HECToR dCSE programme administered by NAG and the ARCHER eCSE programme administered by EPCC.

TPLS solver is highly parallelisable and can simulate flows at ultrahigh resolution ( > 50 million grid points). Key features are:

1. Highly parallelisable using an MPI architecture;
2. Optimised on ARCHER (> 1536 Cores);
3. Fortran coding with PETSc subroutines and parallel-NETCDF I/O.
4. Density contrast flows (liquid/gas flows, density ratios > 1000/1)
5. 3D Domain decomposition and PETSc momentum solvers for efficient parallelism

TPLS 3.0 is available free to use under the BSD Licence and can be obtained from: sourceforge.net/projects/tpls/.