Fast and Massively Distributed Electromagnetic Solver for Advanced HPC Studies of 3D Photonic Nanostructures

Key Personnel

PI/Co-I: Prof. Nicolae Panoiu, Dr. Ahmed Al-Jarro - University College London

Technical: Dr. Mayeul d'Avezac, Dr. Gary Macindoe - University College London

Relevant Documents

eCSE Technical Report: Electromagnetic Solver for Advanced HPC Studies of 3D Photonic Nanostructures

Project summary

At the beginning of the eCSE project, OPTIMET (OPTIcal METa-materials) was a serial code written with somewhat less common and less efficient C++ design pattern and using a large number of dependencies. It had no test suite and used at least three different linear algebra implementations, including an in-house developed set of routines implementing standard matrix-vector multiplication. The work performed in this project built upon our recent development of the OPTIMET code, a project funded by the University dCSE Programme. OPTIMET is based on the multiple scattering matrix (MSM) method, a method widely used in computational electromagnetism. The main computational work in the MSM method consists in computing the system scattering matrix, S and solving a linear system of equations, Sa=b. The vector b is a known quantity (it depends on the parameters of the physical system) and a contains the Fourier expansion coefficients of the scattered electromagnetic field, the main physical quantity that OPTIMET computes. Once a is known, the electromagnetic field can be computed at any point, as well as the corresponding cross-sections. The system matrix is built from block-structures that describe the individual particle geometries, electromagnetic properties and inter-particle interactions.

As part of the eCSE project, the efficiency and functionality of the code were greatly enhanced, as illustrated by the following non-exhaustive list of changes we made:

• OPTIMET was added to GitHub, making full use of its version control feature and issue management system.
• Regression tests were created against the code as it existed on the first day of the project. From there on, unit tests were added during refactoring and when adding new features.
• OPTIMET is integrated into UCL's automated testing infrastructure.
• The dependencies were consolidated to include only libraries with orthogonal features.
• We refactored the code as follows: (i) in-house linear algebra routines were replaced with Eigen; (ii) namespaces were introduced; (iii) RAII rather than systematically multi-step initialization; (iv) smart pointer, STL data structures, Eigen vectors and matrices rather than less robust C data structures; (v) type hierarchy for the major numerical types; (vi) exceptions rather than Boolean returns; (vii) the recurrence computing the translation-addition coefficients was rewritten for simplicity and to remove hard-coded values limiting the applicability of OPTIMET, etc.
• We added a framework to the build system to automatically download dependencies when they are not found on the system.

The improved code opens up the possibility to study a new class of problems in science and technology, problems of significantly increased complexity. Thus, research in photonic metamaterials is now at the crucial point where the knowledge accumulated over the last few years has reached the critical mass that would generate a host of high-impact technological applications. One prerequisite for fulfilling this tremendous potential is the large-scale availability of HPC software tools, which can reduce significantly the costs associated to the research and development of future nano-photonic systems. To this end, we have now made freely available to the academic community working in key areas of science and technology a versatile HPC software tool that rigorously model the interaction between electromagnetic waves and arbitrary, 3D distributions of particles. As is illustrated in the figure above, the improved version of OPTIMET allows one to incorporate in the numerical algorithm key physical effects that significantly broaden the spectrum of applications that can be modelled by the code.

Achievement of objectives

The original proposal had four main objectives:

1. the parallelization of the serial code with the objective of running OPTIMET on HPC;
2. replacing the direct solvers with iterative solvers;
3. the implementation of an accelerated matrix-vector operation via the fast multipole method (FMM); and
4. parallelization of the fast-multipole method over MPI.

During the course of the project we have modified the third and fourth objectives to use a different accelerated matrix-vector optimization scheme than originally envisioned, yet equally efficient. Specifically, we implemented a multiplication decomposed into sparse matrix-vector operations instead, the rotation-coaxial translation decomposition (RCTD). This approach accelerates the calculation without introducing further approximations. Furthermore, an FMM approach can be implemented on top of it in later works, if deemed necessary. All these objectives have been accomplished as such. The code is parallelized using either Scalapack or MPI in the case of the fast matrix-vector operation (RTCD). The linear system can be solved using any algorithm from the Belos package in the Trilinos suite. Furthermore, the sustainability of the code has been greatly increased. The code is now under version control on GitHub. Regression and unit tests were added and the code is under continuous testing on UCL's Jenkins server. A fair amount of the code was refactored to use standard C++ idioms and libraries, while reducing the number of overlapping dependencies and simplifying their installation where necessary.

Summary of the software

OPTIMET is now distributed under the GPL. It is freely available on GitHub for all interested parties. It includes a description of the installation steps, as well as a usage manual. We have installed OPTIMET on ARCHER, including module files to rapidly load it into the path. OPTIMET and the module files are available in the group's shared directories. In addition, the software, the manual and examples of the code can be found at Prof. Panoiu's web page. We plan to make the code even more widely available to the community if possible, through the program library of the Computer Physics Communications journal, via a peer-reviewed journal article.