Grids in grids: hierarchical grid generation and decomposition for a massively parallel blood flow simulator

Key Personnel

PI/Co-I: Dr Rupert Nash (EPCC), Dr Derek Groen (Brunel University London), Prof Peter Coveney (UCL)

Technical: Dr Rupert Nash (EPCC), Dr Derek Groen (Brunel University London)

Relevant documents

eCSE Technical Report: Grids in grids: hierarchical grid generation and decomposition for a massively parallel blood flow simulator

Project summary

Implementing an accurate, consistent, parallel, efficient meshing tool is a very difficult software engineering task. We have created a tool that is accurate and consistent after much effort. Some key parts of the process are parallel and efficient and, for realistic problems, we can now produce voxelisations a factor of ten faster using a 36-core machine.

We have implemeted tools for producing and analysing decompositions and added checkpoint restart capabilities to HemeLB.

Achievement of objectives

The objectives of the work were to:

1. Reimplement HemeLB setup using hierarchical, geometric data structures.
   This has been achieved. The implementation turned out to be extremely challenging and overran significantly. The key issue relates to the difficulty in reliably determining intersections of links between lattice sites and the triangles of the surface. Since these are typically represented with floating-point numbers, intersections are very occasionally missed (or spurious ones created). This can be disastrous for the process as the fluid can then "leak" out of the domain. Adapting the main program to use the new file format was not completed: a simple post-processing step converts from the new to the old formats.
2. Extract HemeLB's domain decomposition into a library (usable from outside the main application) to make the decompositions reusable and the algorithms used configurable.
   This has been achieved, and we now have a standalone library (Protopart+PPStee) which is able to apply and analyse a range of decomposition algorithms. Although not explicitly described in this proposal, we also modified the data structures in the code to bypass domain decompositions in HemeLB. However, this proved to be much more complex than expected, and our initial implementation led to errors for very large problems. In collaboration with researchers at the CCS we have proceeded to work on this after GiG was completed, and incorporated the Zoltan library directly in HemeLB. In this way we can use Protopart+PPStee for quick diagnostics and testing, while applying the optimal algorithms directly into HemeLB using the Zoltan library (which supports the same partitioning schemes).
3. Implement accelerated initialisation of simulations by bootstrapping through one or more coarser discretisations of the domain (using the new hierarchical information).
   This has been partially completed, checkpoint-restart capabilities were added but because we did not finish adapting HemeLB to use the new octree-based format, we could not complete the accelerated bootstrapping easily.
4. Extend the parallelisation of HemeLB setup across multiple nodes using MPI parallelism.
   Not attempted due to time constraints.

Summary of the Software

HemeLB is open source (LGPL licence) and available from https://www.github.com/UCL/hemelb. The software can be easily compiled on ARCHER.

Scientific Benefits

Because of the partial success of the project, the tool as-is will only have moderate impact: improving setup times for our users.