Overview of TOPS – Terascale Optimal PDE Simulations

PIs: V. Akcelik ⁴ , S. Benson ¹ , G. Biros ¹¹ , X. Cai ⁹ , E. Chow ³ , J. Demmel ⁸ , F. Dobrian ⁷ , J. Dongarra ¹¹ , V. Eijkhout ¹¹ , R. Falgout ³ , O. Ghattas ⁴ , B. Hientzsch ⁶ , P. Husbands ² , D. Keyes ⁵ , M. Knepley ¹ , S. Li ^2,8 , T. Manteuffel ⁹ , O. Marques ² , S. McCormick ⁹ , M. Minkoff ¹ , J. Moré ¹ , T. Munson ¹ , E. Ng ² , A. Pothen ⁷ , P. Raghavan ¹¹ , D. Reynolds ³ , R. Serban ³ , B. Smith ¹ , P. Vassilevski ³ , O. Widlund ⁶ , C. Woodward ³ , H. Zhang ¹ ; ISIC affiliates: R. Armstrong (CCTTSS), D. Bailey (PERC), D. Brown (TSTT), P. Colella (APDEC), L. Diachin (TSTT), J. Glimm (TSTT), A. Shoshani (SDM); Applications affiliates: A. Bhattacharjee (CMRS), A. Burrows (SSC), S. Jardin (CEMM), K. Ko (AST), A. Mezzacappa (TSI), H. Najm (CFRFS), R. Sugar (LQCD) ¹ Argonne National Lab, ² Lawrence Berkeley National Lab, ³ Lawrence Livermore National Lab, ⁴ Carnegie Mellon U., ⁵ Columbia U., ⁶ New York U., ⁷ Old Dominion U., ⁸ U. California-Berkeley, ⁹ U. Colorado-Boulder, ¹⁰ U. Pennsylvania, ¹¹ U. Tennessee

Summary

The Terascale Optimal PDE Simulations (TOPS) project is researching, developing, and deploying a toolkit of open source solvers for the partial differential equations (PDEs) that arise application areas such as accelerator design, combustion, fusion, particle physics, and supernovae. Scalable solution algorithms - primarily multilevel methods­ - aim to reduce computational bottlenecks by one or more orders of magnitude on terascale computers, enabling scientific simulation on a scale heretofore impossible. Along with usability, robustness, and algorithmic efficiency, an important goal is to attain high computational performance by accommodating to distributed hierarchical memory architectures.

Nonlinear PDEs give mathematical expression to many core DOE mission applications. PDE simulation codes require implicit solvers for the multirate, multiscale, multicomponent, multiphysics phenomena of hydrodynamics, electromagnetism, chemical reaction, and radiation transport. Problem sizes typically now reach into the tens of millions of unknowns; this size is expected to increase by two more orders of magnitude over the span of the project, assuming hardware procurements track the feasible technical envelope. Moreover, such simulations are increasingly used in parameter identification, process control, and optimization contexts, which require repeated, related PDE analyses.

The convergence rates of solvers traditionally employed in PDE-based codes degrade as the size of the system increases. This creates a double jeopardy for applications – as the cost per iteration grows, so does the number of iterations. Fortunately, the physical origin of PDE problems provides a natural way to generate a hierarchy of approximate models, through which the required solution may be obtained efficiently, by bootstrapping. The most famous examples are multigrid methods, but other types of hierarchical representations are also exploitable, making use of lower fidelity models, lower order discretizations, and even lower precisions. The philosophy that underlies optimality is to make the majority of progress towards a high quality result through inexpensive intermediates.

The efforts defined for TOPS and its collaborations with other projects have been chosen to revolutionize large-scale simulation through incorporation of existing and new optimal algorithms and code interoperability. TOPS provides support for the software packages Hypre, PARPACK, PETSc, ScaLAPACK, Sundials, SuperLU, TAO, and Veltisto, some of which are in the hands of thousands of users, who have created a valuable experience base on thousands of different computer systems.

Most PDE simulation is ultimately a part of some larger scientific process that can be hosted by the same data structures and carried out with many of the same optimized kernels as the simulation itself. By adding a convenient software path from PDE analysis to PDE-constrained optimization, for instance, a preconditioner for the PDE solver becomes a block of a preconditioner for the optimization solver - with reuse of efficiently implemented distributed data structures. Similarly, TOPS increases the value of simulations executed in a fully nonlinearly implicit solver framework by providing sensitivity analysis of the solution with respect to parameters and optimization (parameter identification, control, design, and data assimilation). TOPS linear and nonlinear solvers and time-integrators constitute core technologies for these higher-level tasks (see Figure 1). TOPS has provided users of its Bell-prize winning PETSc toolkit (mainly in fluid flow and E&M for SciDAC applications, but also externally in medical imaging, reservoir modeling, etc.) with all of the solvers in Hypre and SuperLU, without change of the user interface. TOPS is working with SciDAC users to tune its multiparametric solvers to particular applications.

With the CCTTSS ISIC, TOPS is developing standardized solver interfaces relevant to data objects on distributed grids. The PERC ISIC uses a TOPS example to demonstrate their performance analysis and optimization tools. Within TOPS, sparse matrices from SciDAC applications are used in memory hierarchy-based performance improvements. Two important TOPS collaborations that will open the path to many applications beyond those that employ its solvers directly are: ongoing work to support APDEC's adaptively refined meshes and planned work to support TSTT's overlaid meshes and high-order discretizations. Adaptive and/or high-order discretizations are ultimately required by AST, TSI, CEMM, and CMRS, with which TOPS collaborates directly at present using simpler meshes and discretizations.

Figure 1. Inter-relationships between TOPS solver components. In a PDE context, all share grid-based data structures and paral lel software infrastructure. (Arrows indicate dependencies, with scalable linear solvers at the base.)

TOPS co-PIs also perform fundamental algorithmic research in response to known application requirements. For instance, a new form of multigrid relaxation targets Maxwell problems in electromagnetics. A new nonlinear form of preconditioning targets coupled nonlinear problems. Physics-based preconditioning suggested by users is driving development of the nonlinear application solver interface.

The TOPS project webpage may be found at http://www.tops-scidac.org .

For further information on this subject contact:
Prof. David E. Keyes, Project Lead
Columbia University
Phone: 212-854-1120
david.keyes@columbia.edu

back to project page