An autonomous laboratory assistant that uses artificial intelligence (AI) and robotics to cook up new materials has come under fire from researchers who dispute its discoveries.
When scientists unveiled the A-Lab in Nature on 29 November1, they reported that it had produced 41 new materials in 17 days, promising to speed up the discovery of materials that might be used in batteries or electronics, for example.
Google AI and robots join forces to build new materials
But after digging into the A-Lab’s data, some critics say these materials are a mirage. “The paper should be retracted,” says Robert Palgrave, a solid-state chemist at University College London, who critiqued the work in several threads on social-media platform X (formerly Twitter). “The central claim of the paper is that they synthesize new materials, and they have provided nowhere near enough evidence for that.”
The critics also say that the incident is a good reminder that as more and more researchers seek to incorporate AI into their work, they should ensure that the results meet standards similar to those expected of human scientists.
“This paper just reinforces this discussion that we’ve been having about how we make use of AI without running into pitfalls,” says Leslie Schoop, a solid-state chemist at Princeton University in New Jersey.
Gerbrand Ceder, a materials scientist at Lawrence Berkeley National Laboratory (LBNL) and the University of California, Berkeley, who led the A-Lab team, stands by the work. He concedes that the A-Lab’s analysis of its materials might not match a human’s performance, but says the system offers a rapid way to prove that a substance can be made — before human chemists take over to improve the synthesis and study the material in more detail.
Is AI leading to a reproducibility crisis in science?
The A-Lab’s target materials were drawn from the Materials Project, a database at LBNL that includes the structures of thousands of simulated materials. The robot chemist devised ways to make these targets by mixing and heating powdered ingredients, and then analysed the products. Where necessary, it fed these results back into the system so that it could improve the recipe.
The critics celebrate many of the A-Lab’s features, particularly its ability to formulate sensible recipes, and its labour-saving robotics. “I don’t think the entire work is garbage,” says Schoop. “But the analysis of the products clearly failed. Completely.”
The debate over the A-Lab’s data analysis certainly highlights the challenges of using AI in materials discovery, says Andy Cooper, academic director of the Materials Innovation Factory at the University of Liverpool, UK, who has also worked on this kind of automated analysis2. “To me, this doesn’t invalidate the A-Lab concept, but it shows that there is considerable work still to do on the autonomy aspects,” he says.
Diffraction debate
Table of Contents
The A-Lab, which is housed at LBNL, analyses its products using powder X-ray diffraction (PXRD). The technique can reveal how atoms are arranged in the material’s crystalline grains, and help the A-Lab to assess its products’ purity.
To interpret PXRD data, human researchers first create a simulated model of what the material’s crystal structure might be and calculate what X-ray diffraction pattern it would produce. Then they compare that pattern with experimental data and tweak the model until they match. Any mismatches are known as residuals, which indicate how accurately the final model represents the product material. To automate this matching process, the A-Lab uses an algorithm that proved successful in previous papers3,4.
Claimed superconductor LK-99 is an online sensation — but replication efforts fall short
But Palgrave says that the A-Lab’s results contain lots of residuals, meaning that its algorithm is not reliable enough to identify the materials definitively. To demonstrate this, he looked at a series of the A-Lab’s products that were based on lead antimonate, a yellow pigment used by the ancient Egyptians. The A-Lab produced five new materials by swapping some of the pigment’s antimony atoms for metals such as iron, a substitution approach commonly used in materials discovery.
In his X posts on 30 November, Palgrave said that the PXRD data for these compounds were so inconclusive that each sample could actually have been unmodified lead antimonate.
On 2 December, Ceder posted a rebuttal on LinkedIn. He supplied data from a technique called energy dispersive X-ray spectroscopy (EDS), showing that the extra metal atoms had indeed been incorporated into the lead antimonate. These data were not in the original paper because EDS was not part of the A-Lab’s workflow during its first run, Ceder says, although his team is now adding it to the system.
Despite this extra evidence, Palgrave and Schoop say the metal atoms could merely be randomly scattered through the crystals, producing disordered ‘doped’ materials that have been synthesized before and reported in the literature. “We’ve got all the references for them,” Palgrave says.
Ceder maintains that these are new materials, and that the A-Lab’s PXRD analysis is sound. However, he agrees with Palgrave that a couple of the other products that the A-lab ‘discovered’ had indeed been made before by human chemists, and says that he will update the paper to reflect that. He emphasizes, however, that the A-Lab still managed to make these materials without knowing that they already existed: “They were certainly new to the A-Lab.”
A closer look
Palgrave and Schoop have also turned their fire on Nature, arguing that they have found obvious problems that should have been picked up during peer review. The referee reports, which are openly available, suggest that the three reviewers didn’t have much to say about the PXRD data. “That’s a strong failure from Nature,” Schoop says.
Complex molecules made to order in synthesis machine
Palgrave wrote to Nature on 1 December to outline his concerns. “We are aware of criticisms regarding this paper and take all concerns raised seriously,” said Karl Ziemelis, chief physical sciences editor at Nature, in a statement. (Nature’s news team is independent of its journals team.)
Palgrave and Schoop’s teams are now collaborating to check the paper’s claims in more detail and will deliver their findings to Nature. “We are listening to the criticism,” Ceder says, adding that he remains convinced that AI-assisted materials discovery offers a valuable way forward for the field.
Despite the dispute, Palgrave and Schoop agree that this should not undermine the idea of autonomous materials discovery. “I’m very supportive of it — that’s why I care about this,” Palgrave says. “This could be a big game changer for materials synthesis.”