Implementation of multi-level contact detection in granular LAMMPS to enable efficient polydisperse DEM simulations

Key Personnel

PI/Co-I: Tom Shire (U. Glasgow) and Kevin Hanley (U. Edinburgh)

Technical: Kevin Stratford (EPCC)

Relevant documents

eCSE Technical Report: Implementation of multi-level contact detection in LAMMPS

Summary: Summary (pdf)

Project summary

Granular materials occur in many physical and industrial settings, such as geomaterials, avalanches and landslides, crushing of mining ores, food processing and pharmaceuticals. In many cases the grains/granules cover a wide range of particle sizes. For example, the granular filter material used to construct the Bennett Dam in Canada contains particles ranging from 0.08 to 75 mm.

Discrete element modelling (DEM) is a computational tool for predicting how such granular materials will respond during loading, flowing or other processes found in nature or in industry. Each grain in a system is modelled as an individual element, and the elements interact to model to overall system response. The most time-consuming part of a DEM simulation is calculating which grains are in contact with each other and communicating this information between processors when running simulations in parallel. In most DEM codes these stages can become prohibitive when a wide range of particle sizes is employed, as the number of potential inter-particle contacts dramatically increases.

This eCSE project has implemented an improved contact detection scheme for DEM simulations of materials with a wide range of particle sizes into LAMMPS, a widely used open-source software which is especially efficient for massively parallel simulations which can be run using supercomputers such as ARCHER. The new scheme limits the search range over which particle-contacts are checked by defining different ranges depending on the size of particles: e.g., potential contacts between two large particles use a larger search area than between a small and large or two small particles.

For a typical DEM model in which the largest grain diameter is 100 times the smallest, such as that shown below, the newly implemented algorithm allows the simulation to run over 400 times faster and use significantly less RAM memory.

The new implementation also scales well when used in parallel. We are working to incorporate the implementation into the main version of LAMMPS run by Sandia National Laboratories in the USA, which will ensure its long-term sustainability. This new implementation will allow previously intractable problems in materials with a wide range of particle sizes to be simulated using DEM, such as the study of liquefaction of sands, but also will be of interest in other areas such as simulation of colloid/solvent interactions in particle suspensions.

Achievement of objectives

The objectives of the work were to:

1. Adapt existing contact detection in LAMMPS.
   A new multi-level algorithm "neighbor bytype" has been implemented and produces the same results as existing code in LAMMPS. "The same results" here encompasses explicit checks on the number and identity of neighbours, and global measures such as the pair energy for standard test configurations.
   
2. Quantify improvements in simulation time.
   For bi-disperse cases the new algorithm improves performance by up to three orders of magnitude for 100:1 size ratios compared with "neighbour multi".
   For polydisperse cases, improvements of up to a factor of 3 are recorded across a wide range of size ratios compared with "neighbor multi". (Comparisons with "neighbour bin", if practical at all, would be likely to produce very much larger improvements.)
3. Release a well-documented implementation
   While no release is available at the time of writing, we are working with the owners at Sandia to incorporate the changes. Ultimately, the timing of any release in the main LAMMPS code will be determined by Sandia.
   
4. Demonstrate scalability of highly polydisperse DEM simulations on ARCHER.
   For bidisperse cases, the new contact detection algorithm, combined with an improved communication scheme, demonstrates improved scalability on up to 32 (768 MPI tasks) nodes for representative benchmark problems.
   
5. Provide advice to users on the optimum parameters for contact detection.
   A number of common polydisperse cases are considered, and optimum parameters are identified. However, for general particle size distributions, some caution is warranted in making sweeping statements.

Summary of the Software

LAMMPS [1] is a flexible and popular code performing classical molecular dynamics. It is developed by Sandia and is publicly available under GPL [2]. While the developments made by this work are not yet incorporated into the main version of LAMMPS, we hope they will be so in the near future. In the meantime, the relevant updates can be made available on request.

[1] https://lammps.sandia.gov
[2] https://github.com/lammps/lammps

Scientific Benefits

We believe the improvements represent a potentially very significant increase in the efficiency of the LAMMPS code in cases where there is a large size discrepancy in the particles in the system. Such systems are common in granular media, and will allow new kinds of system to be studied in appropriate detail.