SciDAC Institutes SciDAC Partnerships Basic Energy Sciences Biological and Environmental Research Fusion Energy Sciences High Energy Physics Nuclear Physics Participating Orgs SciDAC Overview (2000-2011) Logos DOE Office of Advanced Scientific Computing Research (ASCR)

TOPS Scalable Solvers for Complex Field Simulations

PIs: G. Biros ¹⁰ , X. Cai ⁹ , E. Chow ³ , F. Dobrian ⁷ , V. Eijkhout ¹¹ , R. Falgout ³ , O. Ghattas ⁴ , B. Hientzsch ⁶ , D. Keyes ⁵ , M. Knepley ¹ , S. Li ^2,8 , T. Manteuffel ⁹ , S. McCormick ⁹ , A. Pothen ⁷ , D. Reynolds ³ , R. Serban ³ , B. Smith ¹ , P. Vassilevski ³ , O. Widlund ⁶ , C. Woodward ³ , H. Zhang ¹ ;
ISIC affiliates: P. Colella (APDEC), L. Freitag (TSTT), P. Hovland (PERC), L. McInnes (CCTTSS);
Application affiliates: A. Bhattacharjee (CMRS), A. Burrows (SSC), J. Chen (CEMM), E. D'Azevedo (TSI), K. Germaschewski (CMRS), S. Jardin (CEMM), A. Mezzacappa (TSI), H. Najm (CFRFS), R. Samtaney (CEMM) ¹ Argonne National Lab, ² Lawrence Berkeley National Lab, ³ Lawrence Livermore National Lab, ⁴ Carnegie Mellon U., ⁵ Columbia U., ⁶ New York U., ⁷ Old Dominion U., ⁸ U. California-Berkeley, ⁹ U. Colorado-Boulder, ¹⁰ U. Pennsylvania, ¹¹ U. Tennessee

Summary

In support of SciDAC astrophysical, combustion, and fusion simulations, the Terascale Optimal PDE Simulations (TOPS) project is creating a new generation of solvers for PDE field problems.

Partial differential equations (PDEs) arise in core DOE science missions in a variety that defies any general-purpose computational approach. For dense and sparse linear systems and ordinary differential equations, the DOE has developed very successful, broadly applicable numerical libraries . For PDE systems, it has instead developed national laboratories, where reside code groups with application-specific expertise in modeling, discretization, solution, and interpretation of simulation results.

TOPS aspires to provide a general-purpose toolkit of scalable solvers for the systems that arise in inner loops of the vast majority of these special-purpose codes, in which PDE field equations are reduced to large systems of nonlinear or linear equations by local discretizations (finite differences, finite elements, finite volumes) on Eulerian grids. Solving such systems commonly consumes 50-90% or more of the execution time of PDE-based applications in SciDAC.

TOPS has focused to date on SciDAC astrophysics, combustion, and fusion applications in order to demonstrate success and to find stimuli to address end-to-end software issues that might be overlooked in models that abstract only the mathematical difficulties. These are multirate, multiscale, multicomponent, multiphysics applications.

Figure 1. Target applications for TOPS are multirate, multiscale, and multicomponent, such as the Hall reconnection MHD problem (c/o CMRS, now released as PETSc ex29) shown here at early and late times.

Being multirate , they require implicit solvers to “step over” dynamically irrelevant but stability-limiting fast waves. Multiscale implies fine grids, either uniformly or adaptively. As multicomponent problems, they inherit blocking that carries special implications for data structures and cache locality. As multiphysics problems, they are often approached through operator splitting. Scalable software solutions developed under TOPS do not ignore the investments embodied in legacy solvers, but seek to incorporate them as preconditioners. This is accomplished by keeping code as data structure-neutral as possible, using callbacks, and regarding vector-to-vector transformations as the fundamental tasks.

TOPS presents multiple levels of interfaces to applications. Users who wish to pass matrix elements to the solver may do so, and take advantage of both traditional iterative methods and the algebraic multigrid (AMG) solvers of the H ypre library . Users who supply more discretization and grid information can take advantage of solvers at the next level.

For purposes of sensitivity analysis and optimization, TOPS encourages users to migrate to a yet higher level and interface with a Jacobian-free Newton-Krylov (JFNK) solver, for which they supply pointers to a subroutine that is called to evaluate a high-fidelity nonlinear residual and to a second subroutine to precondition its Jacobian (which need never be explicitly computed). These subroutines can be created using automatic differentiation during the build phase. For time-dependent problems, TOPS solvers are callable by the user on each user-governed timestep, or the user may call an adaptive implicit integrator, which calls a rootfinder on each timestep.

Eventually, many SciDAC applications codes may not call TOPS solvers directly at all. APDEC and TSTT software may perform the discretization and, in turn, call TOPS for implicit solves of subsystems, such as scalar Poisson problems on AMR grids, or multicomponent blocked nonlinear problems on overlaid grids.

TOPS supports the CEMM fusion code M3D through a matrix-element interface, using AMG-GMRES to solve linearized, operator-split unstructured scalar problems on 2D poloidal crossplane grids at each timestep. This exploits PETSc-Hypre interoperability. M3D's previous domain-decomposed iteration was adequate for today's problem sizes but does not scale to fine grids. TOPS, APDEC, and CEMM have collaborated by incorporating the KINSOL Newton-Krylov solver into a resistive MHD demonstration code. Investigations focus on JFNK to solve a fully implicit formulation of magnetic reconnection problems on a uniform grid. Extensions to AMR are planned.

With TSI, TOPS collaborates at the level of coupled field linear problems from AGILE-BOLTZTRAN, using an operator-split preconditioner with separate phases of point-first and component-first orderings. These problems, with large blocks due to multigroup radiation, are unidimensional. For the 2D and 3D future of TSI, TOPS' parallel multigrid solvers are needed.

CMRS and TOPS are collaborating on Hall magnetic reconnection, for which TOPS has demonstrated JFNK, and will evaluate the benefit of fully implicit temporal integration. TOPS expects eventually to take all three codes to 3D implicit formulations with high-order discretizations on adaptive grids.

The TOPS project webpage may be found at http://www.tops-scidac.org .

For further information on this subject contact:
Prof. David E. Keyes, Project Lead
Columbia University
Phone: 212-854-1120
david.keyes@columbia.edu

Reports