MedMeta: Testing LLMs on Evidence Synthesis, Not Recall

Author-review draft. Source-grounded working copy; not approved for publication. Metrics are quoted only where the source states them.

Medical benchmarks that reward factual recall have become saturated. A different question remains open: can a model read several studies and write a defensible combined conclusion? That is the gap MedMeta targets.

MedMeta is a benchmark I built with Benoit Favre and François Portet. It contains 81 meta-analyses from PubMed, published between 2018 and 2025. Each item supplies only the abstracts of the studies cited in the original analysis, and asks a model to produce a conclusion comparable to the meta-analysis authors'.

Two evaluation workflows

The benchmark evaluates two distinct workflows. In the Retrieval-Augmented Generation (Golden-RAG) setting, the model receives the ground-truth abstracts. In the Parametric-only setting, it relies solely on its internal knowledge.

The headline finding is that the grounded workflow consistently and significantly outperforms the parametric one across models. Even with a favorable retrieval setup, models perform better when the relevant source material is actually provided than when they are asked to recall it from memory.

The paper also finds that domain-specific fine-tuning brings only marginal benefits, and that those benefits are largely neutralized once external material is provided. That second result matters for anyone choosing between building a specialized model and building a better retrieval pipeline.

How the evaluation was validated

To support comparisons at scale, MedMeta uses an LLM-as-a-judge protocol. The paper reports two checks against human expert ratings: a high Pearson correlation (r = 0.81) and a Bland-Altman analysis showing negligible systematic bias. Those two checks together support treating the automated judge as a reasonable proxy for expert scoring in this benchmark's setting.

I want to be precise about what that validation does and does not cover. It supports using the judge for this evaluation workflow. It does not turn the judge into a general-purpose grader for any medical text, nor does it replace expert review where stakes are high.

The limit that deserves more attention than the score

Under ideal retrieval conditions, current LLMs still score only slightly above average, around 2.7 out of 5.0. A reader could stop there and conclude that models are simply not good enough yet. The stress tests suggest a more specific problem.

All models tested, regardless of architecture, failed to reliably identify and reject negated evidence. In other words, when a study's conclusion contradicts or negates the expected finding, the models did not consistently catch that reversal. For clinical summarization, that failure mode is arguably more important than the headline score. A confident summary that misses a negated result is worse than an uncertain one, because it hides its own error.

The paper's own conclusion follows from this: for clinical applications, developing robust RAG systems is a more promising direction than model specialization alone. The evidence for that claim is the combination of the two workflow results: grounding helps consistently, while specialization helps marginally and mostly disappears once sources are provided.

What I take from it

MedMeta is a deliberately hard benchmark, and its value lies in what it makes visible. The workflows separate what a model knows from what it can do with provided evidence. The judge validation makes scalable evaluation defensible. And the negation failure gives builders a concrete failure mode to test for, rather than a vague sense that medical reasoning needs improvement.

Primary source: MedMeta: A Benchmark for LLMs in Synthesizing Meta-Analysis Conclusion from Medical Studies, arXiv:2605.09661.