libcfd2lcs: A general purpose library for computation of Lagrangian coherent structures during CFD simulation
eCSE04-014Key Personnel
PI/Co-I: Dr. Justin Finn - University of Liverpool, Dr. Romain Watteaux - University of Cambridge, Dr. Andrew Lawrie - Bristol University
Technical: Dr. Justin Finn - University of Liverpool
Relevant Documents
eCSE Technical Report: libcfd2lcs Technical Report
Project summary
Several diagnostics have been proposed in recent years that allow fluid transport and mixing patterns to be objectively defined in time dependent or chaotic (turbulent) flows. One such family of diagnostics, coined Lagrangian Coherent Structures (LCS), correspond to the locally most attracting or repelling material surfaces in a flow. They have been instrumental in developing new understanding of transport in a variety of geophysical, biological, and industrial applications. However, the resource intensive post-processing procedure typically used to extract these features from experimental or numerical simulation data has probably limited the impact of LCS to date.
In this eCSE project, a new computational library, libcfd2lcs, was developed, that provides the ability to extract LCS on-the-fly during a computational fluid dynamics (CFD) simulation with only modest additional overhead. Users of the library create "LCS diagnostics" through a simple but flexible API, and the library updates the diagnostics in the background as the user's CFD simulation evolves. By harnessing the large-scale parallelism of platforms like ARCHER, this enables CFD practitioners to make LCS a standard analysis tool, even for large scale three-dimensional simulations. Moreover, libcfd2lcs provides access to the time evolution of LCS with unprecedented detail, giving researchers a powerful new tool to study unsteady flow structures.
libcfd2lcs has a flexible and easy to use API, which is accessible from C and Fortran. The library has already been integrated with several flow solvers, allowing research groups to add LCS capability to their analysis toolbox without significant new code development or post-processing time. Care was taken to ensure good single core performance, and tests on ARCHER have demonstrated good parallel scaling of the library to over 1000 computational cores.
Achievement of objectives
Create a general purpose parallel library, libcfd2lcs, to compute Lagrangian Coherent Structures (LCS) during CFD simulation. This library will have all the functionality of the original implementation, but without dependency on a particular flow solver, and only minimal external dependencies. The library will be designed to treat the user’s flow solver as a “black box” that need only provide velocity fields for the LCS computations, making it broadly useful to the ARCHER CFD community. The library will be documented and installed as a public module on ARCHER.
By all measures this objective was achieved. The library exceeds the capabilities of the original implementation and is much more user friendly. It has been successfully utilized by at least 4 (very) different flow solvers to date. It is installed on ARCHER and documented in its own user manual.
Demonstrate good scalability to over 1000 cores: Specifically, a strong scaling efficiency of 80% or greater for the LCS related computations will be targeted for up to 1024 compute cores.
The library has been tested using between 1 and 4096 computational cores on ARCHER. The parallel performance of the library is detailed in the Technical Report. When I/O costs are excluded, the library demonstrates greater than 80% strong scaling efficiency from 1-1024 cores.
Develop broad usage of libcfd2lcs by the ARCHER user base: In order to facilitate widespread use of our library, we aim to familiarize ARCHER users with basic LCS theory and the possibilities of using LCS in their own research. In addition, we will solicit suggestions from the ARCHER CFD community for how to make the library more usable for them "off-the-shelf".
At the time of writing, we know of at least 6 research groups in the UK and abroad who are working with libcfd2lcs to add LCS capabilities to their CFD simulations. Engagement with colleagues, especially at the UK National Oceanographic Centre (NOC), throughout the development of the library led to implementation of several important new capabilities that should encourage new users of the library. In June, 2016, a webinar was given through the ARCHER technical forum, which introduced users to LCS as an analytic tool, and demonstrated the steps for integrating the library into their own applications.
Demonstrate a relative computational overhead of 15% or less of the total simulation time.
This metric depends very strongly on the speed of the flow solver used to perform the CFD simulation and the type of LCS computed. A "typical" simulation using one of three flow solvers discussed in the Technical Report required a libcfd2lcs overhead of 4-44%. Considering that these three flow solvers are significantly faster than the one used in the original implementation, we believe we were successful in developing a "fast" implementation, but perhaps too ambitious in our objective metric.
Summary of the software
libcfd2lcs is a general purpose numerical library that performs the calculations needed to extract Lagrangian Coherent Structures (LCS) from time dependent flows. The library is based on the integrated approach described by Finn and Apte (2013) for computing finite time Lyapunov exponent fields. It is designed to work with two or three dimensional hydrodynamic datasets produced by computational fluid dynamics (CFD) simulations or experiments. It can interface directly with a distributed memory CFD application, allowing for parallel, on-the-fly calculation of LCS. It is primarily written in modern Fortran, but has a fully functional C interface. It can perform parallel I/O using the HDF5 standard.
libcfd2lcs is distributed under the GNU public licence and is available to all ARCHER users as a public module. The full source code, example programs, and documentation are available online.
