Office of Science Logo Skip Naviagation

SCaLeS

Science Case for Large-scale Simulation

Breakout Session Abstracts

Accelerator Modeling

Astrophysics

Biology

Chemistry

The nearly 70-year old DuPont advertising slogan “better living through chemistry” has evolved over the years[1] but still encapsulates the significance of our discipline. Chemistry is perhaps the most impactful and pervasive of all the physical sciences. It affects and controls our daily lives in innumerable and underappreciated ways, and is central to the current and future well-being of ourselves, our economy, and the ecology of the entire planet. Examples include:

In all of these endeavors, the understanding and manipulation of molecular processes is essential.

Historically, chemistry has been a predominantly experimental science with a strong theoretical foundation derived from quantum theory, statistical mechanics, and classical mechanics. The exponential growth of computing power over the last three decades, coupled with increasingly sophisticated theoretical and numerical methods, has enabled computer simulation to emerge as a junior partner to experiment. Only a junior partner, because the predictive capabilities of computation are still limited to either small or idealized systems. Large areas of chemistry have not yet succumbed to simulation. For instance, heterogeneous catalysis, protein chemistry, the function of many enzymes, soot formation, and nano-scale devices all elude quantitative description.

Looking to the future, it is clear that computation will play an increasingly important role because of the expected continued growth of computer power. However, computers are not just getting faster; they are also getting cheaper. The day is not too far off when, if it is possible to simulate something, then to do so will be cheaper, faster, and have less environmental impact than the corresponding experiment. Furthermore, theory and simulation can answer questions that cannot be posed or solved experimentally. When fully integrated into a research or design process, interactive, quantitative simulation can dramatically accelerate timescales by changing the manner in which science is conducted.

The chemistry panel must address the following questions and pose additional ones:

How do we achieve this vision of chemistry becoming a computational science supported by experiment? If we do nothing will it still happen? What are the impediments to this? What are the consequences of it? What should we be investing in now? What do we as computational and theoretical chemists need to be doing differently? What are compelling examples of what might be accomplished a decade or more from now? What is the timescale on which we can expect change to be accomplished?

One answer is clear – if theory and computation are to emerge as an equal partner to experiment, let alone become the senior partner, then the investment in simulation should become commensurate with that presently made in experiment.

[1] http://heritage.dupont.com/touchpoints/tp_1939/overview.shtml

Climate/Earth Science

The Climate and Earth Science session will be held offsite in conjunction with the Community Climate System Model Workshop in Breckenridge, CO. The importance of climate and earth system science to provide sound scientific and predictive information about how the climate system varies has been noted in the June 6, 2001, National Academy report titled, "Climate Change Science: An analysis of some key questions." Specifically identified is the need to "reduce the wide range of uncertainty inherent in current model predictions of global climate change" by producing advances in "the understanding and modeling of the factors that determine atmospheric concentrations of greenhouse gases and aerosols and the so-called 'feedbacks' that determine the sensitivity of the climate system to a prescribed increase in greenhouse gases."

Each increment of computing power permits reduction of model uncertainties through the inclusion of additional physical processes or through more accurate and realistic solutions of existing processes. For example, the ability to routinely run climate simulations in which atmospheric chemistry and aerosol models, ocean biogeochemical models, and terrestrial ecosystems models interact will lead to fully coupled carbon cycle-climate simulations. Simulations at higher resolutions will permit better representations of topography and small-scale eddy processes in the ocean and will lead to better predictions of climate at regional scales. The breakout session will address the steps necessary for improving climate simulations and how those improvements will be enabled by advances in computational science and technology.

The mathematical methods of interest include: semi-implicit and operator split methods to allow long-time integrations in the presence of fast-moving gravity waves, Lagrangian vertical coordinate systems and conservative remapping schemes that allow accurate thermodynamic simulation of transport and moist processes, and fast methods for solution of elliptic systems. Also of considerable interest are software engineering practices that allow a community model to develop amongst hundreds of computational climate scientists, mathematicians, computer scientists, and numerical analysts.

Combustion

Environmental Remediation/Processes

Materials Science

Plasma Science

Plasmas are collections of charged particles (electrons and ions) and is sometimes called the 4th state of matter. The plasma state is ubiquitous in the universe. Plasmas tend to shield-out slowly varying electric fields, but they interact strongly with magnetic fields, both externally imposed and created by electrical currents in the plasma itself, and will both absorb and reflect electro-magnetic fields.

Magnetic fusion is based on creating a high-temperature, fusing plasma of heavy-hydrogen and confining it with a donut shaped magnetic field, called a tokamak or a stellarator. Analysis is required in order to calculate the efficiency of heating and driving current in the plasma with externally applied radio-frequency (RF) electro-magnetic fields, of predicting the fine-scale turbulence that leads to anomalous diffusion across the confining magnetic fields, and to calculate the stability of the gross configuration against deformations that destroy it. For each of these applications, different mathematical descriptions of the plasma are used, ranging from solving the equations of motion of individual particles to treating the entire configuration with magneto-fluid equations. The current trend is in integrating these different analyses.

Heavy-ion fusion builds on particle accelerator technology in order to make an efficient and cost effective driver for inertial fusion. Here the idea is to converge intense energetic-ion particle beams in a symmetrical way to implode a small pellet of heavy hydrogen. The plasma science problem is to maintain a quality beam of sufficient energy and density, from the time the ions are created through the acceleration and focusing process, and the subsequent interaction with the solid target. This are makes heavy use of the particle-in-cell (PIC).

The Earth-Sun plasma system is dominated by the fueling from the solar wind, and the confinement provided by the Earth's magnetic field. DOE interest in this stems from the effects of this plasma on communication systems and space travel, as well as on the opportunity to study fundamental physical processes by using satellite measurements.

QCD

Quantum Chromodynamics (QCD) is the component of the Standard Model of high energy physics that describes the strong interactions. The Standard Model has been enormously successful in describing matter at the shortest-length scales to which it has been probed. However, our knowledge of it is limited because it is extremely difficult to extract many of the most important predictions of QCD, those which depend upon the long-distance behavior of the theory. At present, the only means of doing so is through large-scale numerical simulations within the framework of lattice gauge theory. Such simulations are necessary to solve the fundamental problems in high-energy and nuclear physics that are at the heart of the large, experimental efforts in these fields. Participants in this breakout session will identify the major calculations that must be addressed in the next few years in order to provide support to the experimental programs in a timely fashion and to keep pace with theoretical physicists in Europe and Japan. They will also discuss long-term goals.

Participants will discuss the hardware, software, and human resources needed to meet both the near- and long-term objectives of the field. They will identify barriers to achieving these objectives--and the means of overcoming them. Lattice QCD calculations have a number of simplifying features not present in many high-end computing applications, such as regular grids, uniform, predictable communications between processors, limited memory, and I/O requirements. Participants will discuss how to take advantage of these features to construct computers for the study of QCD that are far more cost effective than general-purpose computers. They will identify the challenges to developing software for special-purpose computers and the progress that is being made to meet these challenges. The development of new algorithms and calculational techniques is as important to lattice QCD simulations as are improvements in hardware performance and in software. Participants in this session will identify recent algorithmic advances and major challenges.

CFD/MHD

Discrete Mathematics

Many aspects of large-scale parallel simulations rely critically upon discrete mathematics and discrete algorithms. The purpose of this breakout session is to articulate these dependencies and to identify the research that will be needed to enable the kinds of simulations that DOE needs to perform in the years ahead.

Discrete algorithms play many roles in parallel scientific computing. Large parallel machines require efficient mechanisms for message routing, scheduling, and data management--all of which reduce to combinatorial problems. Graph partitioning and coloring are key tools in the parallelization of scientific computations. Geometric discretization and meshing techniques are central to many scientific computations and involve sophisticated geometric algorithms. The solution of linear and nonlinear systems is facilitated by advanced graph algorithms. This is widely recognized in sparse direct methods, but graph algorithms are also key kernels of many algebraic preconditioners. Jacobian and Hessian construction can be made much more efficient by the use of graph coloring methods. And matroids play a role in the construction of sparse bases for numerical optimization. Numerous graph problems also arise in automatic differentiation. Advances in biology rely critically on sophisticated string algorithms to assemble, search and analyze genomic data. Other aspects of the biological sciences also depend upon combinatorial algorithms. Proteomics includes advanced geometric algorithms for structural comparisons; phylogenetics relies upon fast algorithms for tree enumerations and branch-and-bound algorithms; microarray analysis exploits advances in learning theory. Learning theory also plays a growing role in the analysis of computational experiments.

In all these areas and many more, discrete mathematics plays an essential role in large-scale scientific simulation. The breakout will try to define and prioritize the areas in which discrete math and algorithms are most urgently needed to advance DOE's scientific computing agenda.

Solvers and "Fast" Algorithms

Traditionally, applications have gained as much speedup from algorithms as they have from computer hardware. Indeed, in many instances, the speedup due to improved algorithms exceeds the speedup due to improved hardware. Such enhancements have come mainly through the discovery of clever new algorithms that have good mathematical characteristics, exploit special properties and structure in the applications, and take advantage of the strengths of existing computer systems. Currently, the trend in applications is toward multi-disciplinary, multi-scale, high-resolution codes. This trend will have an impact on existing algorithms. Increasing complexity of the physical processes being simulated and orders of magnitude increases in problem size will render today's algorithms ineffectual. Thus, it will become critical to adapt existing approaches or invent new algorithms to address these new applications. The "Solvers and Fast Algorithms" breakout session will consider these issues. We will attempt to identify applications where improved algorithms will enable important breakthroughs in science, either by permitting the solution of problems of greater size or finer scale than current technology allows, or by enabling the solution of simulation problems currently considered intractable. We will examine the barriers to the development of these scalable solvers and algorithms, and suggest the computing and human resources needed to overcome the barriers.

Charter

The scalable solution of large complex systems on parallel computers is crucial for achieving overall performance of application codes. By scalable we generally mean the ability to use additional computational resources efficiently to solve increasingly larger problems. In some cases the notion of scalability can be thought of as having two basic components: algorithmic scalability and implementation scalability. Both components can be difficult to achieve on today’s massively parallel computer systems. In addition, questions of efficiency and robustness must be addressed in the implementation of good algorithms.

At the heart of many simulation codes lie solvers for systems of linear and nonlinear equations. In many cases, a substantial fraction of the computational time in a simulation is spent within either a nonlinear or linear solver. For this reason, it is critically important to develop and implement solvers that scale both to the size problems envisioned and to the tens of thousands of processors expected to be employed.

Possible questions to address in the breakout session:

Meshing Methods

The numerical solution of PDEs is pervasive in the compuational science research community. One critical component of these solution techniques is the use of advanced meshing technologies to better approximate the physics of interest. For example, discretizing complex geometrical domains using sophisticated mesh generation techniques is often the first step in the solution process. As the simulation proceeds, dynamic adaptive procedures such as h- or r-refinement and front tracking are used to automatically improve solution resolution. Furthermore, in the DOE computing environment, it is imperative that these techniques perform well on massively parallel computers through effective distributed memory algorithms and dynamic load balancing methods.

In this breakout session, we examine existing advanced meshing techniques in the context of the needs articulated in the application breakout sessions. We will identify areas where the use of existing technologies could result in high impact to these applications and determine the new advances needed in our own area which can potentially revolutionize the way computational problems are solved. We will identify the barriers to these advances (hardware, algorithmic, and software) and the resources required to overcome them.

The technology areas covered in this session are broad and include:

Breakout Session co-leaders:

Lori Freitag Diachin
Sandia National Laboratories
ladiach@sandia.gov

David Serafini
Lawrence Berkeley National Lab
dbserafini@lbl.gov

David L. Brown
Lawrence Livermore National Lab
dlb@llnl.gov

Multi-physics Solution Techniques

The focus of the multi-physics solution techniques breakout session is on accurate, robust and efficient numerical methods for solving coupled, nonlinear, time dependent systems. Such systems typically exhibit multiple time-scales that require at least one equation/subsystem to be treated implicitly. Invariably these methods lead to large nonlinear and linear algebraic systems. For this reason the accurate, robust and efficient solution of these problems is very challenging.

Typical multi-physics applications might couple, for example: fluid mechanics, solid mechanics, thermal energy transfer, mass transfer with chemical reactions, electro-magnetics, radiation transfer, particle transfer, etc. The numerical methods of interest might include: operator split, sub-cycling, semi-implicit, or fully-implicit time integration methods. The nonlinear solvers might include fully-coupled, fixed-point or decoupled solution strategies. For exploration of complex nonlinear solution spaces these methods can also be complemented with parameter continuation, bifurcation and stability techniques. Additionally, in order to provide truly predictive multi-physics simulation capabilities there will be an increased need to understand the generation and propagation of numerical error through these coupled solution techniques.

In this math breakout session we will review the material presented in a number of the scientific application sessions of the first day, in light of the mathematical methods described above. A standard set of questions developed by the SCaLeS organizing committee for cross-cutting solution technologies will be used for the breakout session. These questions essentially assess the requirements of the applications, the availability of appropriate mathematical algorithms and software, the estimated level of effort to advance the current simulation capability, and the required hardware resources to move to this new level of computational simulation capability. Results and vision from this discussion will be contributed to the larger workshop report.

Breakout Session co-leaders:

Dana Knoll
Los Alamos National Laboratories
nol@lanl.gov

John N. Shadid
Sandia National Laboratories
jnshadi@sandia.gov

Multiscale Techniques

Radiation Transport and Kinetic Methods

Problems requiring fast, robust, and accurate solution of the radiation transport equation and its approximations occur in such widely varying fields as astrophysics, atmospheric physics, fusion, and medicine. The radiation transport and kinetic methods breakout session will focus on the deterministic and stochastic numerical methods for solving the transport equation and its variant approximations (e.g diffusion equation). Topics we will consider include:

We also will consider the role of validation and verification in helping to assess the quality of the solutions generated by the various algorithms.

Since radiation transport occurs in a variety of multiphysics contexts, we also will consider algorithmic choices for the coupling of transport to other physics.

Uncertainty Quantification (including statistics)

Uncertainty quantification deals with the determination of the difference between a real outcome from the physical world and the predicted outcome from the computational/mathematical model world. The real and predicted outcomes can be expected to differ for several reasons:

The grand challenges for uncertainty quantification involve the development of statistical and probabilistically based methodology that can address the uncertainty sources; account for the many small errors that can add to create a big, adverse effect; address the multivariate nature of the problem; look at both deterministic and stochastic models; and incorporate the scientific complexity of the problems. Straightforward solutions of this problem tend to be computationally prohibitive so a more imaginative approach will be necessary. Our ability to develop a comprehensive approach to uncertainty quantification becomes more difficult as the complexity of the phenomena being modeled grows and the complexity of the hardware evolves.

Architecture

Ultrascale simulations by their very nature require the most powerful computing systems. The computational power of these systems has been increasing at a tremendous rate. However, these systems remain a challenge to use effectively. Physical limits related to the speed of light and engineering challenges such as power dissipation lead to complex tradeoffs between the architecture, the algorithms, and the programming used to express the algorithms. This session will consider four principal areas.

  1. Expectations for Mainstream Computer Architecture

    Most ultrascale simulation today is conducted on systems that heavily leverage commodity processor and interconnect technology. Of the Top500 systems, virtually all (494) are clusters, combinations of clusters (constellations), or massively parallel processors (MPPs) that leverage commodity processor technology. Of the top 25, 9 are clusters using primarily commodity components, including 3 of the top five. This situation is a result of the rapid advance in the power of commodity processors, driven in part by the need to meet the Moore's law goal of a doubling in performance every 18 months.

    However, these commodity systems are not designed for ultrascale computing. For example, they have typically not supported the memory bandwidths and latency hiding techniques needed by many scientific calculations, leading to low achieved performance as a fraction of peak. On the other hand, these systems, because they exploit the commodity market, can benefit from economies of scale, leading to relatively high performance per dollar ratios. In fact, many recent winners of the Gordon Bell prize for price-performance have been clusters of commodity processors. Other areas that have not advanced at the same pace as the CPU include interconnects between nodes and I/O performance. What can the ultrascale community expect from the commodity market?

  2. Supporting Current Algorithms and Applications

    Current applications represent an important resource; these are built up over the years and contain many person-years of development. Their behavior (and idiosyncrasies) are often well-understood. Providing better architectural support for these applications has a clear near-term benefit. What are the greatest challenges faced by current applications with respect to current computer architecture? What are the needs in networking, I/O, memory systems, CPUs, and interconnects?

  3. Supporting New Algorithms and Applications

    Historically, the benefit of new algorithms has often been as great as the rapid advance in computer hardware. For example, the improvement in the speed of solution for sparse systems of linear equations was roughly as great as the improvement in computer system performance over the same time period. Further, these new algorithms, by seeking optimality in terms of floating-point operations, often reduce the amount of data reuse, greatly stressing the memory systems of current architectures. Adaptivity in algorithms also stresses current designs, by decreasing the predictability of instructions and memory references. How will the needs of new algorithms change the design points for hardware in the areas of CPU, memory systems, I/O, interconnects, and networking? What is the potential impact of other programs, such as DARPA's High Productivity Computing Systems initiative?

  4. Disruptive Technologies

    A fresh look at the tradeoffs between architecture, algorithms, and programming models has the potential to produce a qualitatively different solution. Quantum and molecular computing are two such "disruptive technologies", though their impact is decades away at best. Are there other possibilities for changing the approach to ultra-scale computing? What technologies offer the greatest promise? But any great change must be justified by great benefit, particularly given the rapid and continuing improvement in the performance of commodity processing.

Access and Resource Sharing

Data Management and Analysis

Terascale computing has the potential to revolutionize the way science will be conducted. More accurate and more sophisticated simulations will facilitate new scientific discoveries not possible with current technology. However, a potential barrier to the benefits expected from terascale computing is the large volume of data and the complexity of the data generated. Without attention to technology that can facilitate the annotation, management, flow, and the efficient archiving and retrieval of the data generated, the potential of terascale computing cannot be achieved.

The Data Management and Analysis breakout session will discuss issues related to technologies for moving, storing, querying, and manipulating massive amounts of complex data that will be generated in terascale computing environments. The primary purpose of such technologies is to achieve highly efficient storage, search, and access of the massive amounts of data expected. However, another important goal is to enable scientists to concentrate on their science while minimizing or eliminating completely the need to deal with tedious tasks related to data storage, file movement, and management of other computer resources.

The panel, consisting of both application scientists and data management experts, will describe the data management problems they are already experiencing in their specific application domains. They will then try to predict how these problems will be affected by an increase of 100-1000 fold in computational capabilities and propose solutions and research directions. Some questions that the panel will address are listed below:

  1. What are the data bottlenecks you are currently experiencing and what data management problems slow you down in your scientific exploration?
  2. How would the nature of your exploration change with the data volumes produced with terascale computing?
  3. What are the data rates your simulation programs generate today in the data generation, post processing, and analysis phases, and how will these be affected by the several orders of magnitude increase in computational capability under terascale computing?
  4. What data management technologies currently are used in your application, and what new or advanced technologies will be required because of terascale computing?

Examples of technologies to be considered include:

Frameworks and Environments

This is a fairly broad subject area and the participants of this breakout group will begin by identifying some critical areas for a more focused discussion. Rather than discussing individual compiler, runtime systems, or libraries by themselves, participants will be asked to take a more holistic approach and discuss the higher-level question of how such components can be incorporated to significantly improve the programmers' experience.

Generally we consider frameworks to be something in which user software is "contained.” From the ancient CLOS and Smalltalk environments, the word "framework" is derived from the "frame-based style" of programming promoted by these early object-oriented languages.  An environment in this context is a programming model--i.e., a language or class of languages--and the associated tools used for developing software.  The Gnu C++ compiler is not a programming environment, but the Java language with its virtual machine and RMI communication is.  A framework is an environment that also provides some structure: user-provided software elements that may have different roles related to the application they are implementing. A component framework is an environment with an additional model for composing components into applications. A programming language for parallel computing, for example, may not be construed to be a framework because it imposes no structure, but nonetheless is an environment in its own right. There likely is a lot of dissent about these particular definitions.

The purpose of this session is to focus on the programming techniques that are likely to be most beneficial for future machines and applications, with enough well-defined terminology that those techniques can be described. All frameworks and environments serve to partition complexity. In some cases, environments make parallel computing seem like serial computing, by parallelizing user code under the covers. In other cases, they exist only to provide a standard for composing applications out of various user-provided software elements, such as component architectures. In every case, however, these frameworks and environments serve to partition the complexity of an application between user and environment provider or between users and other users.

Questions for the Frameworks and Environments (F&E) Breakout Session

Performance Tools and Evaluation

Software Management and Support

State of the art scientific simulations often require very large and complex codes that typically develop and evolve over a period of a decade or more. Increasingly, these simulation codes are collaborative efforts that span many institutions and areas of expertise. Climate modeling is one community already facing the issue of developing, maintaining, evolving, documenting, and distributing large community simulation codes. By contrast, the lifetime of a computer is typically only three years. The scientists and software developers as well as the resulting codes may thus span three or four machine generations.

Simulation is a fundamental method for research; accordingly, we must ensure that scientific software meets the same standards of practice as do design and planning of laboratory instrumentation and field or lab experiments. Thus, the testing, validation, and execution of all components become critical, especially in a diverse and distributed computing environment. The commercial sector, where development of complex software systems is a driving force, employs software engineering practices that are not routinely used in scientific research environments. There is thus the need to evaluate approaches to software development to determine the degree to which they are applicable to the development of high performance models. The software engineering approaches to be considered cut across all areas of software development including tools, processes, structure, and standards. Some of the approaches to be considered include:

Success in this arena will depend on developing a software management plan and paradigm for software development that simultaneously manages software complexity, scales to large numbers of processors, provides reliable tools, and supports the scientific process (i.e., code is open to peer review, experiments are reproducible, and results are verifiable) while maintaining computational efficiency.

System Software

Visualization

The term "scientific visualization," coined in the landmark 1987 ACM Computer Graphics survey article, [2] refers to the process of transforming abstract data into images for the purposes of gaining insight. Visualization provides the means to "see" data, which is a fundamental analysis activity performed across all scientific disciplines. The challenges facing modern visualization span many different dimensions and are exacerbated by the increasing size and complexity of data created by modern science programs.

Following the SCaLeS charge and mission statements, the Visualization breakout session aims to provide answers to the questions posed for the CS breakouts as a whole:

  1. Identify the application (science) areas that can benefit from new visualization methods, deployments and approaches.
  2. Quantify "success."
  3. Identify resources needed to achieve the goals asserted in (1) and (2).
  4. Identify barriers between where we are now and goals from (1) and (2).

In the Visualization breakout session, five speakers will present brief, 5-minute position statements that will focus on providing some starting points for discussion by the breakout as a whole. Each speaker's presentation will focus on directly addressing the four questions above.

Visualization topics germane to the SCaLeS mission include, but are not limited to:

  1. Moving towards "standards" within the visualization community to achieve software tool interchange and reuse across a wide variety of science domains. This broad topic includes fostering closer ties with related communities having similar goals, such as Scientific Data Management.


  2. Evolution of visualization software architecture to accommodate the general trend towards distributed collaborative science teams, as well as diverse computing platforms.


  3. Evolution of visualization algorithms beyond scalar field visualization to provide support for large, multivariate and time varying data.


  4. Clarifying the relationship between visualization, which is often a post-processing activity, and computing/simulation. Specifically, computing and simulation is often performed in batch mode, while visualization is often performed interactively. The result is a dramatically different set of approaches for resource management that often are in conflict with each other.


  5. Defining a "business model" for visualization programs that provides a path from research prototype to production software that receives funding for ongoing support, maintenance, and training.


  6. Identifying strategies to influence hardware and software vendors. Specifically, visualization and graphics is "held hostage" by the gaming community, and the trend in recent years has been towards low-cost, high-performance triangle engines. While programmable GPUs are interesting, they fall short in many key areas, such as providing hardware support for advanced graphics techniques like Image Based Rendering. The graphics hardware vendors are motivated by economics, and scientific visualization is a small market compared to the gaming market.


  7. Defining vehicles or instruments that allow direct requirements input from science programs to guide visualization research and development activities.

The chairs of the Visualization breakout session ask that you come equipped with pictures and paragraphs to contribute to the report.

[2] McCormick, B.H., T.A. DeFanti, M.D. Brown (ed), Visualization in Scientific Computing, Computer Graphics Vol. 21, No. 6, November 1987.