Discretization Tools for Terascale Computational Science from the Terascale Simulation Tools and Technology (TSTT) Center

PIs: J. Glimm ^1,2 , D. Brown³ , L. Freitag³
Co-PIs: E. D'Azevedo⁵ , P. Fischer⁶ , P. Knupp⁴ , X.L. Li² , M. Shephard⁷ , H. Trease⁸
Affiliated Researchers: J. Drake⁵ (Climate), K. Ko⁹ (Accelerators), S. Jardin¹⁰ (CEMM), D. Quinlan³ (PERC ISIC)

¹ Brookhaven National Laboratory, ² State University of New York at Stony Brook, ³ Lawrence Livermore National Laboratory, ⁴ Sandia National Laboratories, ⁵ Oak Ridge National Laboratory, ⁶ Argonne National Laboratory, ⁷ Rensselaer Polytechnic Institute, ⁸ Pacific Northwest National Laboratory , ⁹ Stanford Linear Accelerator Center, ¹⁰ Princeton Plasma Physics Laboratory.

Summary

A major goal of the TSTT Center is to deliver interoperable discretization software for PDE-based terascale scientific simulation. Through both algorithm and software development, we are delivering high-order time and space discretizations and boundary conditions to application scientists for use in mesh-based simulation codes. SciDAC-enabled collaborations with the fusion, high-energy accelerator modeling, and climate modeling communities provide an early insertion path for our technology and assure the relevance of our interoperable discretization tool development.

Diffusion of an initially isolated Gaussian pulse following the magnetic field lines in the Tokamak using spectral element discretizations.

Vision. Many simulation projects in the Department of Energy use the solutions of partial differential equations (PDEs) to model physical behavior of interest. Solving these PDEs on terascale computers first requires the representation of the equations and their solution at discrete points in space and time. TSTT discretization technology is being used to impact SciDAC application efforts through direct insertion of currently available TSTT algorithmic and software technology into the application solution process and through the development of new techniques that will impact future software development. In particular, we are working with domain scientists in fusion and accelerator design to enable scientific discovery through the application of state-of-the-art discretization technology.

Fusion. Our work with the fusion community has targeted the use of high-order and adaptive discretization technologies. In collaboration with fusion scientists, we are testing spectral element and adaptive finite element technologies on problems of relevance to MHD fusion .

Spectral Element Discretizations: The spectral element (SE) method offers geometric flexibility, rapid numerical convergence, and efficient data reuse for cached-based architectures. Pictured here are SE simulation results of anisotropic diffusion in a model Tokamak geometry. Tokamaks and related configurations are being considered by the DOE Center for Extended MHD Modeling (CEMM) as possible fusion reactor configurations. A desired feature of the reactors (and of the simulations) is to have minimal diffusion in the toroidal cross-section with respect to the axial diffusion. These systems can be difficult to model accurately because the ratio of diffusion coefficients in the two directions can be as high as 10 9 . Results for three different spectral element mesh cross sections showed that for a fixed resolution, high-order approximations are superior to low-order in every case.

Adaptive Finite Element Discretizations: W e have developed an adaptive finite element solution procedure to solve incompressible magnetohydrodynamic (MHD) flow problems formulated in stream function-vorticity form. We use a stabilized finite element formulation and adaptive h -refinement based on a posteriori temporal and spatial error estimates of the magnetic field. We are working with scientists from CEMM to apply this adaptive procedure to a tilt instability problem that consists of two oppositely directed currents embedded in a uniform magnetic field. This configuration is unstable and so the magnetic vortices turn, align horizontally, and repel each other. As the vortices align horizontally, thin current sheets form, which are difficult to capture without adaptivity. The results shown in the figure below illustrate the current sheets and adaptive computational mesh when the current sheets have formed.

We will investigate the use of discontinuous Galerkin (DG) methods for MHD applications that offer several advantages relative to traditional finite element method. In general, they conserve physical quantities on an element level, have compact schemes that are easy to parallelize, and simplify both adaptive h- and p -refinement. With MHD problems, they avoid excessive diffusion and have properties similar to spectral element methods (which may make them easier to incorporate into the M3DP code).

High-energy Accelerator Modeling . Essential to the design of advanced high-energy physics accelerators is the ability to simulate and model the electromagnetic wave-guide cavity structures that steer and focus a beam of high-energy charged particles through the accelerator. Performing this task requires the computer solution of the time-dependent Maxwell's equations of electromagnetics. In collaboration with scientists at SLAC, we are analyzing and developing new discretization schemes for solving these equations to increase the robustness of their simulation and design tools.

In particular, we are investigating the use of higher-order finite elements for spatial discretization and higher-order symplectic time integration. These methods have significantly less numerical dispersion and are more accurate than the currently used DSI methods. In addition, they are inherently stable requiring no dissipation or filtering techniques. We have used this method on a SLAC-provided mesh of an accelerator cavity and demonstrated a stable and accurate solution. We are currently working on implementing the boundary conditions and current/voltages sources that are required for wakefield calculations. In future work, we will develop error estimators for our H(curl) finite element method to provide accuracy information for the simulation and semi-orthogonal finite element basis functions to significantly reduce the run time for higher-order simulations.

Development of interoperable discretization tools. Within the TSTT center, we are leveraging our broad expertise in finite difference, finite element, and spectral element discretization methods for partial differential equations (PDEs) to provide interoperable discretization tools that will enable the rapid development of new software applications for scientific discovery. In support of this long-term goal, researchers in the TSTT center have been separating and re-implementing low level discretization operators from their respective frameworks in preparation for insertion into an interoperable discretization library. Research in collaboration with the PERC ISIC will lead to automatic performance optimizations

Further information: http://tstt-scidac.org

Contact Information:
James Glimm, Brookhaven National Laboratory
Phone: 631-333-8155
glimm@bnl.gov

David L. Brown, Lawrence Livermore Nat. Lab.
Phone: 925-424-3557
dlb@llnl.gov

Lori Freitag Diachin, Lawrence Livermore Nat. Lab.
Phone: 925-422-7130
diachin2@llnl.gov

back to project page