Illustration credit: Kyle Palmer/ PPPL Communications. CC BY.

PPPL researchers find overoptimism in journal articles using AI machine learning to solve fluid-related partial differential equations.

The hype surrounding machine learning, a form of artificial intelligence, can make it seem like it is only a matter of time before such techniques are used to solve all scientific problems.

While impressive claims are often made, those claims do not always hold up under scrutiny. Machine learning may be useful for solving some problems but falls short for others.

Researchers at the US Department of Energy’s Princeton Plasma Physics Laboratory (PPPL) and Princeton University performed a systematic review of research comparing machine learning to traditional methods for solving fluid-related partial differential equations (PDEs).

Such equations are important in many scientific fields, including the plasma research that supports the development of fusion power for the electricity grid. 

The researchers found that comparisons between machine learning methods for solving fluid-related PDEs and traditional methods are often biased in favour of machine learning methods. They also found that negative results were consistently underreported.

They suggest rules for performing fair comparisons but argue that cultural changes are also needed to fix what appear to be systemic problems.

Ammar Hakim, PPPL’s deputy head of computational science and the principal investigator on the research, said: “Our research suggests that, though machine learning has great potential, the present literature paints an overly optimistic picture of how machine learning works to solve these particular types of equations.”

Partial differential equations

PDEs are ubiquitous in physics and are particularly useful for explaining natural phenomena, such as heat, fluid flow and waves, and scientists and engineers have developed various mathematical approaches to solving them.

One approach is known as numerical methods because it solves problems numerically, rather than analytically or symbolically, to find approximate solutions to problems that are difficult or impossible to solve exactly.

Recently, researchers have explored whether machine learning can be used to solve these PDEs. The goal is to solve problems faster than they could with other methods.

The systematic review found that in most journal articles, machine learning hasn’t been as successful as advertised.

Lead author Nick McGreivy, who recently completed his doctorate at the Princeton Program in Plasma Physics, said: “Our research indicates that there might be some cases where machine learning can be slightly faster for solving fluid-related PDEs, but in most cases, numerical methods are faster.”

Numerical methods have a fundamental trade-off between accuracy and runtime.

McGreivy added: “If you spend more time to solve the problem, you’ll get a more accurate answer. Many papers didn’t take that into account in their comparisons.”

Furthermore, there can be a dramatic difference in speed between numerical methods.

In order to be useful, machine learning methods need to outperform the best numerical methods, yet McGreivy’s research found that comparisons were often being made to numerical methods that were much slower than the fastest methods.

Two rules

Consequently, the paper proposes two rules to try to overcome these problems.

The first rule is to only compare machine learning methods against numerical methods of either equal accuracy or equal runtime. The second is to compare machine learning methods to an efficient numerical method. 

Of 82 journal articles studied, 76 claimed the machine learning method outperformed when compared to a numerical method.

The researchers found that 79% of those articles touting a machine learning method as superior actually had a weak baseline, breaking at least one of those rules.

Four of the journal articles claimed to underperform when compared to a numerical method, and two articles claimed to have similar or varied performance.

McGreivy said: “Very few articles reported worse performance with machine learning, not because machine learning almost always does better, but because researchers almost never publish articles where machine learning does worse.”

Hakim said: “There’s a lot of hype in the field. Hopefully, our work lays guidelines for principled approaches to use machine learning to improve the state of art.”

To overcome these systemic, cultural issues, Hakim argued that agencies funding research and large conferences should adopt policies to prevent the use of weak baselines or require a more detailed description of the baseline used and the reasons it was selected.

He said: “They need to encourage their researchers to be sceptical of their own results. If I find results that seem too good to be true, they probably are.”

The new paper is published in Nature Machine Intelligence.