PNNL

After four years of development, including significant contributions from Pacific Northwest National Laboratory scientists and data engineers, a new Earth modeling system—the Energy Exascale Earth System Model (E3SM)—is now available to the broader scientific community. The U.S. Department of Energy (DOE) Office of Science supports the E3SM project through the Office of Biological and Environmental Research.

Those involved with E3SM sought to overcome previous modeling limitations by taking advantage of the latest high-performance computing technologies. With this capability, the new high-resolution, coupled Earth system model will help researchers explore the challenges posed by interactions of weather-climate variability with energy and related sectors.

"The mechanisms of our planet are very complex," said E3SM Chief Scientist L. Ruby Leung, a PNNL atmospheric scientist and Battelle Fellow. "We want to represent how interactions of atmosphere, oceans, land, and ice govern the behavior of Earth as a system and predict future changes in water supply, extreme events, and coastal vulnerability. With this information, we can inform decisions to help strengthen the resilience of populations, our economy, energy resources, and national security."

PNNL scientists have had critical roles in E3SM's progress:

• As the project's chief scientist since 2016, Leung is responsible for the scientific direction of E3SM, setting priorities for development of the coupled model and its components, and defining the science questions and modeling experiments of simulation campaigns.
• Phil Rasch, Laboratory Fellow and chief scientist for climate science at PNNL, is co-lead of the E3SM Atmosphere Group and a co-principal investigator of the E3SM project at PNNL.
• Kate Calvin, research scientist, co-leads the E3SM Land/Energy Group. Calvin is based in College Park, Maryland, at the Joint Global Change Research Institute, a collaboration between PNNL and the University of Maryland.
• Susannah Burrows, atmospheric scientist, leads the coupled biogeochemistry simulation group to perform modeling experiments that address the E3SM biogeochemical cycle questions.

Their involvement and the work of other PNNL scientists and engineers is described below.

Connecting Land and Energy

PNNL researchers worked extensively on incorporating water processes into E3SM. They developed a river model that simulates streamflow and inundation over land to provide freshwater input to the ocean. This model is coupled to a water management model, also developed by PNNL, to represent reservoir operations and water use.

Researchers also developed new ways of representing irrigation, including how crops are irrigated, where the water is drawn from, and how often irrigation occurs.

Meanwhile, additional PNNL researchers focused on representing human activities related to energy infrastructure and how these activities can affect the biogeochemical cycle. Overall, the E3SM Land/Energy Group focused on the water and biogeochemical cycles and how nutrient limitations, such as those of nitrogen, influence carbon uptake. The team also looked at feedbacks between what the land can produce, such as forestry and food, and what people need.

PNNL contributors in the Land/Energy Group are Ben Bond-Lamberty, Kate Calvin, Maoyi Huang, Guoyong Leng, L. Ruby Leung, Hongyi Li, Xiangyu Luo, Zeli Tan, Teklu Tesfa, Nathalie Voisin, and Tian Zhou.

Adding Details About the Atmosphere

E3SM's atmosphere component started from a well-known model called the Community Atmosphere Model, a part of the Community Earth System Model project. PNNL contributed to many improvements in developing the E3SM atmosphere component, particularly to clouds, turbulence, and aerosols (tiny particles in the air), and by substantially increasing the model's vertical resolution from 30 to 72 layers.

"We recognize that many features in the atmosphere have very strong gradients—they go from large values to small values very quickly," Rasch said. "In order to accurately represent those variations in fields such as water vapor, clouds, and aerosols that often vary greatly with altitude, you need model layers that are quite thin. And so we made the decision to go to thinner layers than have historically been used in atmospheric Earth system models, particularly near the surface."

However, the representations of clouds, turbulence, and convection didn't work as well with the new layers at first. These processes operate at smaller scales in nature than a model can easily represent, so approximations need to be introduced to allow them to be included. PNNL had a key role in understanding how these features and processes interact, and why they are sensitive to vertical resolution. They worked to identify how these approximate and uncertain aspects could be adjusted and how to calibrate—or retune—the parameters so the model could reproduce observed atmospheric features more realistically.

Researchers also pushed the model boundary much higher in the stratosphere and improved aerosol formulations and the treatment of clouds—particularly ice clouds—in the model.

"At this point in the game, the atmosphere model represents a significant improvement in the climate over the previous generation model," Rasch said.

Other primary PNNL contributors in the Atmosphere Group are Kaylee Brent, Susannah Burrows, Dick Easter, Steve Ghan, Bryce Harrop, Po-Lun Ma, Yun Qian, Balwinder Singh, Hui Wan, Hailong Wang, Jin-Ho Yoon, and Kai Zhang. Burrows, Ma, and Singh received separate DOE awards for their outstanding contributions on the project.

Capturing and Storing the Data

Another PNNL group was responsible for capturing model information and simulation history (data provenance) and helping to ensure that simulations could be easily reproduced. The team developed a tool to collect large-scale scientific application process history, simulation configuration information, and result files.

"On E3SM, we identified some of the key provenance information from E3SM simulations and collected it for each simulation," said Bibi Raju, a data architectures scientist. "After a simulation run, the simulation's provenance can be harvested and stored in a database, where it can be searched and recalled later by other modelers to easily reproduce a simulation using the same configuration and host environment."

Final simulations are stored on an international grid managed by the Earth System Grid Federation. PNNL is working with this international consortium to publish provenance data as well, "providing data users a level of transparency required by researchers who require high-integrity products," Raju said.

PNNL staff also made important strides in integrating new codes and improving computational performance so the model could run more efficiently.

In addition to Raju, PNNL contributors in this group are Todd Elsethagen, Tim Shippert, Balwinder Singh, and Eric Stephan.

A Multi-Institutional Team Effort

E3SM will reliably simulate aspects of Earth system variability and project decadal changes that could critically affect the U.S. energy sector in the near future. Scientists will use it to help answer overarching scientific questions associated with the water cycle, biogeochemistry, and cryosphere (continental and ocean ice) systems.

The project involves more than 100 scientists and engineers at multiple DOE national laboratories and universities. The national laboratories are PNNL, Argonne, Brookhaven, Lawrence Berkeley, Lawrence Livermore, Los Alamos, Oak Ridge, and Sandia. E3SM also includes contributions from several DOE programs.

Involvement in E3SM has given scientists and engineers an opportunity to learn about other areas that they might not be exposed to on their own.

"I really like the breadth of the project and the interdisciplinary nature," said Calvin, whose research focuses on integrated human-Earth system modeling. "The multilab aspect has been really fun, to learn not just about my own research area, but also other aspects of the land and about ocean and atmosphere."

For more information and a video on E3SM, see the news release, "PNNL scientists contribute to new, high-resolution earth system model," or visit the E3SM website.