LLMs will provide accurate climate info, except when they won’t





Make no mistakes

“You’re right”—LLMs will get climate info wrong if you ask

Some models fare better, but all fail in similar ways.


Scott K. Johnson

–


|

5






Story text








Many studies over the years have examined how people encounter climate science information. Some focused on the top results that Google surfaced (back when Google was still in the business of serving links), while others examined social media content or newspaper and TV news coverage. With large language models (LLMs) now forming a share of the public information diet, it’s worth looking at how they handle climate change.

But there’s no direct way for researchers to look at the information people have seen. It is served ephemerally to individual users, confined to their chat session. A study led by researchers from University College London and the United Nations Educational, Scientific, and Cultural Organization (UNESCO) tried to tackle this question from multiple angles, testing LLMs with different prompts and evaluating their output. They started with benchmarks of simple factual questions (benchmarks that might even be in models’ training data at this point) but also analyzed more complex interactions—including when users think they’re right to push back on an answer.

CO2 Q&A

LLM research often faces (no-true-Scotsman-tinged) criticism for focusing on a particular model rather than some other frontier model that may have been released after the experiment concluded. In this case, the study used 33 models, including multiple versions of the major frontier and open-weight models. The flagship frontier models used were Claude Fable 5, Claude Opus 5, Gemini 3.1 Pro Preview, Gemini 3.6 Flash, and GPT-5.6 Sol. (You can find the full list in the report.)

All the models were scored for correct answers to 600 questions from several benchmark sets. This included multiple-choice textbook questions, “reading comprehension” questions based on text snippets from scientific papers, and questions about how confident climate scientists are in different conclusions drawn from IPCC reports.

That last one was the most difficult, with the flagship models answering around 78 percent correctly and the smallest open-weight models coming in at around 47 percent. For the other benchmarks, frontier models and large open-weight models generally scored around 90 percent, while small open-weight models scored in the 70s.

Two persistent trends show up here: The latest frontier models tend to score a bit better than earlier generations of that model, and larger open-weight models score better than small ones.

But the researchers didn’t just test for accurate factual recall; they also prompted models to evaluate whether evidence supported or refuted specific claims. (For example, the models were asked whether the statement that “renewable energy investment kills jobs” is accurate, given evidence like descriptions of employment and investment statistics.)

Here, the best models scored around 70 percent accuracy, while the worst models scored below 40 percent.


Bar chart of model performance recalling facts and judging evidence. Text reads: Larger, newer AI models do better, and the gap between knowledge and reasoning is wider for smaller models.

The 33 models are grouped into categories. The open-weight models are listed by the number of parameters (in billions).

The 33 models are grouped into categories. The open-weight models are listed by the number of parameters (in billions).


Credit:

Drobnjak, et al./UNESCO

Finally, the researchers tested 656 questions about the potential impact of policies on global temperatures. Here, they could simply run each scenario in a common simulation tool and then compare its output to each LLM’s answer.

Some questions asked whether a single policy (like a ban on nuclear power plants) would increase or decrease future warming—those were answered correctly 72 to 89 percent of the time. Others asked how a pair of policies would combine—those were answered at a rate no better than random guessing. Asking how much a policy would increase or decrease future warming returned even worse results.


Chart of scores on different types of policy questions. Text reads: AI model’s accuracy falls step by step as climate policy questions get more complex.


Credit:

Drobnjak, et al./UNESCO

Tell me what I wanna hear

Another set of experiments using 15 of the models evaluated what happens when you argue with the LLM. The researchers tried more than responding “nuh-uh” or “no it isn’t.” They also tried claiming to be a specific kind of person.

With the user identifying as either a climate scientist, a policy adviser, or a climate skeptic, they asked LLMs to judge the accuracy of some common false climate claims like “CO2 is not the main driver of climate change.” At first, that claimed identity had no impact on the likelihood of the LLM judging the statement correctly.

But then the researchers started pushing back. Responding by confidently restating the false claim as if it were true was moderately successful for “climate scientists” but not for “climate skeptics.” But presenting sustained disagreement and counter-arguments over several rounds caused all 15 models to relent to a significant degree, reducing their accuracy by 30 to 35 percent.


Chart of results for the three identities, and the three different tests. Text reads: Who the user claims to be matters little; what they assert, and how persistently, matters a lot.


Credit:

Drobnjak, et al./UNESCO

Sycophancy is a familiar issue for LLMs, which will bend output to be agreeable to the user’s prompts. It didn’t take much pushing in these tests, either. For the vast majority of the cases in which the LLM changed its answer, it did so at the first of the three rounds of pushback, when the user simply responded, “I don’t think that’s right.”

Watch your language

One more set of experiments checked to see how the models performed in languages besides English. The researchers translated three of the factual recall benchmarks into 12 other languages. They chose some that are well represented on the Internet (like Spanish and French) and some that should have a smaller footprint in training data (like Urdu and Swahili). For this test, they dropped down to just five models.

The average accuracy score dropped in non-English languages, but most were relatively close. The top six only dropped by about two percentage points or less. But at the lower end, scores dropped by seven (Urdu) or even 10 percent (Swahili). And about half the questions got differing answers from a given model depending on the language.

Some models fared better than others. GPT-5.6 Sol only scored about 3 percent lower in Swahili compared to English, for example, while DeepSeek-V3 fared 16 percent worse.


Chart title text reads: Some AI models are more multilingual than others: accuracy lost from English to Swahili. GPT-5.6 Sol scored -2.7, Claude Opus 5 scored -7.3, Nova Pro scored -12, Gemma 3 12B scored -12.9, and DeepSeek-V3 scored -16.3.

The results for other languages fall in between English and Swahili, so this gives a sense of the spread in each model.

The results for other languages fall in between English and Swahili, so this gives a sense of the spread in each model.


Credit:

Drobnjak, et al./UNESCO

Use at your own risk

The researchers didn’t interpret these results entirely negatively. “[Frontier models] can answer most textbook-style climate questions correctly and can produce open-ended explanations that climate scientists judge comparable to PhD-level work,” they wrote. “But this knowledge is uneven: models are less reliable at judging evidence and scientific uncertainty; that knowledge is not equally accessible across languages; it is not always robust to user pressure; and models fail dramatically when asked to reason about how climate policies interact.”

Using LLMs for climate information, then, requires sticking to questions they perform reasonably well on and avoiding questions they’ll confidently answer despite being no more connected to reality than a random guess would be. That’s quite a lot to ask of users—especially when LLMs are presented as tools that can do anything you ask.

Zooming out to the big picture, users are certainly asking LLMs all these kinds of questions (and more). This study gives us a window into the information those users are receiving.

Photo of Scott K. Johnson


Scott K. Johnson

Associate Writer
Scott has written about geoscience and energy at Ars as a freelancer since 2011.


5 Comments

Leer artículo original en Ars Technica