What is physics-informed machine learning?

Machine learning is a branch of artificial intelligence and computer science that focuses on the use of data and algorithms that attempt to imitate the function of the human brain, improving in accuracy over time.

Machine-learning algorithms use statistics to find patterns in large amounts of data, including numbers, words, images, clicks, or other relevant data. If it can be digitally stored, it can be fed into a machine-learning algorithm, according to the MIT Technology Review.

Deep neural networks—their internal structure resembles the neurons in our bodies, hence the name—are the backbone of deep learning algorithms. These man-made information messengers take in input and provide output.

To learn the parameters that go into the network, scientists complete a process called training, using data they already know.

This can best be illustrated in trying to teach a computer to recognize an image as either a cat or a dog. To do this, scientists input thousands of images of both animals in every imaginable shape, size, and color, calibrating the parameters of the network.

It is through this process that the machine learns how to take us from an image to a verdict.

For them to provide the most accurate predictions—to not, for example, confuse a white Birman cat with a mid-sized shih tzu—they need a tremendous number of samples on which to train.

And that is where physics-informed machine learning comes in.

The connection between the input and the output is called a map, and it can be applied to provide predictions in various complex system settings, including weather forecasts.

If, for example, a researcher wanted to predict the temperature in New York City for a given day, they might input weather-related data for many days prior. The more information, the better the prediction.

Such a task might require a staggering number of samples. Physics-informed machine learning changes this: in this case, it would call upon prior physical models of the temperature distribution in New York over time.

People have been modeling physical systems for hundreds of years. Physics-informed machine learning allows scientists to use this prior knowledge to help the training of the neural network, making it more efficient. This means it will need fewer samples to train it well or to make the training more accurate.

It should be noted that in this case, scientists don’t simply provide the neural network with information—they constrain the data according to the physical model the data should satisfy.

Constraint is a key concept when it comes to physics-informed machine learning. The input is not random data points but, in the case of the weather example, part of the physical process for which we already know the laws or equations.

The practice has multiple benefits. Physics-informed machine learning allows scientists to cut the number of training samples, in some cases by several orders of magnitude.

Physics-informed machine learning background

Machine learning has been around for decades. No one knows for sure the first time it was used, but its history is often recounted with several key events. A 1943 paper by logician and cognitive psychologist Walter Pitts and American neurophysiologist Warren McCulloch attempted to mathematically map out thought processes and decision-making in human cognition. The two provided a way to describe brain functions in abstract terms and showed that simple elements connected in a neural network can have immense computational power.

Seven years later, in 1950, English mathematician Alan Turing developed his “Turing Test,” meant to determine if a computer has real intelligence. According to Turing, the question whether machines can think is itself “too meaningless” to deserve discussion. Instead, he reasoned, it was worthwhile to determine whether a digital computer can do well in a certain kind of game that Turing describes (“The Imitation Game”). In this case, a remote human interrogator within a fixed time frame must distinguish between a computer and a human subject based on their replies to various questions. According to Turing, a computer’s success at “thinking” can be measured by its probability of being misidentified as the human subject.

Two years later, Arthur Samuel wrote the first computer learning program, the game of checkers. The IBM computer used in this experiment improved its performance the more it played, showing it had, in fact, learned. Later that same decade, in 1957, Frank Rosenblatt designed the first neural network for computers—the perceptron—which simulated the thought processes of the human brain.

In 1981, Gerald Dejong introduced the concept of “Explanation Based Learning,” in which a computer analyzed training data and created a general rule it could follow by discarding unimportant information. A decade later, work in this area underwent an important transformation, shifting from a knowledge-driven approach to a data-driven approach. It was then that scientists started creating programs for computers to analyze large amounts of data and draw conclusions from the results.

In 2006, Geoffrey Hinton coined the term “deep learning” to explain new algorithms that let computers “see” and distinguish objects and text in images and videos. Five years later, IBM’s Watson beat a human competitor on the TV trivia show Jeopardy!.

The year 2011 saw the invention of Google Brain. Its deep neural network was able to discover and categorize objects, and shortly after it was developed, the system had taught itself to recognize cats.

Machine learning is key to the development of self-driving automobiles. Although these vehicles are not yet part of our everyday lives—there are far fewer of them on the road than was predicted even five years ago—they are still a subject of great fascination and a focus of major car manufacturers.

Despite this delay in the automotive industry, machine learning is already inside our homes. It’s what is at work behind online recommendations—think the products pushed on Amazon and shows promoted by Netflix—and is critical in fraud detection.

Physics-informed machine learning debuted in the 1990s, appearing in scattered papers throughout the decade. A resurgence in machine learning around 2010 breathed new life into this promising offshoot.

Another milestone came in 2014 with Facebook’s DeepFace. Its algorithm was able to detect whether two faces in unfamiliar photos are of the same person with 97.25% accuracy, despite differing lighting conditions or angles. Humans generally have an average of 97.53% accuracy, meaning Facebook's facial-processing software had nearly the same accuracy as a human being.

Physics-informed machine learning importance

Physics-informed machine learning has the potential to allow scientists to attack problems they could not address before.

It can lead to the design of better drugs by helping reduce the size of the design space and by guiding the choice of new experiments.

Perhaps one of its greatest promises is in better prediction of fluid flow, which appears almost everywhere, from blood vessels to river waters. Fluid flow is also a key concern for the oil and gas industry—physics-informed machine learning can help in this endeavor by improving the simulation of fluid flows in complex geometries, which traditionally have been very expensive.

Strengths and limitations of physics-informed machine learning

One of the greatest strengths in physics-informed machine learning is that it yields results quickly—in just a fraction of a second. Because of the flow structure of the neural network, the output of a new sample is very efficient. And when the training is successful, the predictions are impressively accurate.

But the technology is not without limitations. If researchers don’t have enough samples to train it with, it won’t train well. And, in many physical settings, acquiring enough samples can be prohibitively expensive

While we might have millions of pictures of dogs and cats with which to train our systems, we have less information about, for example, some chemical substances.

And even when samples are abundant, it still might not be enough to prevent catastrophe in high-stakes situations. Self-driving cars make moment-to-moment decisions based on billions of samples. But that still isn’t enough to eradicate all potential fatalities.

For self-driving vehicles to be safe, they need an enormous amount of training data. Ideally, they should be fed with billions of hours of footage of real driving. But this is more difficult than one might think. Not only is it costly, but some events—including those surrounding accidents or the identification of roadside debris—are relatively rare.

This means a self-driving car might not have experienced all iterations of driving before hitting the road. Car makers are acutely aware of this deficiency and have tried to account for it. Yet the predictions just a few short years ago that self-driving cars would be on our roadways in numbers by 2020 did not materialize.

Because physics-informed machine learning is trained on samples, it doesn’t extrapolate well on samples that are not somehow familiar to it. The reason why? Neural networks are very good interpolators, but not good extrapolators.

Imagine training a neural network using a collection of samples derived under similar conditions. In this case, the neural network predictions will likely be very good—they are very good function approximators, and they can represent accurately high-dimensional functions. But if information is provided to that same network from samples that were derived under different conditions, the results can be unpredictable or wrong.

Scientists cannot, for example, train a neural network to recognize cats and dogs by only providing the network with a single image. These systems need millions of images to make solid predictions. This can be a problem for all sorts of machine learning, including in the creation of pharmaceutical drugs.

A neural network can be successfully trained to recognize a collection of drugs and make accurate predictions about its effectiveness. But this same system will be far less effective when it comes to predictions about drug compounds, because it cannot successfully predict anything about drugs with which it is unfamiliar.

Future applications of physics-informed machine learning

In addition to the development of new materials and compounds, physics-informed machine learning may make tremendous strides in personalized medicine.

Imagine having a tool that could design a drug—or drug therapy—to meet the needs of a single patient based on that person’s family history and medical needs.

Personalized medicine, an emerging field of study, uses an individual's genetic profile to guide decisions regarding the prevention, diagnosis, and treatment of disease.

Scientists know a substantial portion of variability in drug response is genetically determined, with age, nutrition, health status, environmental exposure, and other therapies influencing drugs’ effectiveness.

Physics-informed machine learning could help by accounting for factors no single doctor—or even a team of doctors—might discover. Properly trained neural networks can identify problems no human being could detect on their own.

Those networks can achieve that by accounting for correlations among variables that are too complex for a human to identify. This property can be further improved by enhancing neural network training with counterfactual arguments.

Its role in designing better experiments is promising. Scientists can ask machines which type of study they should conduct in order to reach a certain outcome. The machines often craft experiments no human could have imagined—and they end up being better.

Physics-informed machine learning at Pacific Northwest National Laboratory

Pacific Northwest National Laboratory (PNNL) has been working in this area for years. That work has culminated in recent insights with implications for developing medicines and industrial systems controls, among other areas. In May 2021, scientists at PNNL announced findings in relation to a study examining a pair of graph generative models for the therapeutic design of novel drug candidates targeting SARS-CoV-2 viral proteins.

PNNL’s research was timely and could accelerate drug discovery in future pandemics.

In April 2020, scientists at PNNL embedded prior knowledge about molecular structure in a neural network to facilitate the identification of molecular water configurations with preferrable properties. This kind of ability helps increase the interpretability of deep neural networks for chemical applications and can facilitate their adoption by the broader community.

In that same month, PNNL announced strides in the area of differential equations, which are frequently used in engineering domains such as modeling and control of industrial systems, where safety and performance are of the utmost importance.

PNNL also made gains in the area of subsurface modeling, presenting in July 2020 a physics-informed neural network approach that needs fewer measurements to estimate both the state and the parameters of fluids flowing in the subsurface. This research can help, among other efforts, clean up at the Hanford Site, a decommissioned nuclear facility in the state of Washington.

It was around that same time that the lab announced a new means of incorporating measurements contaminated by noise into the training of physics-informed neural networks. Using a framework based on probability theory, scientists proposed an extension of physics-informed neural networks to quantify the effect of noisy measurements on state estimation and system identification for various problems. This can be particularly helpful in using real-world measurements, which always contain some noise.