PhD defense Rajaa El Hamdani: Measuring Parametric Knowledge in Large Language Models: Datasets, Benchmarks, and Decoding Protocols
Télécom Paris, 19 place Marguerite Perey F-91120 Palaiseau [getting there], amphi 3 and in videoconferencing
Jury
- Antoine Doucet, Full professor, University of La Rochelle, France (Reviewer)
- François Yvon, Senior Researcher, CNRS, France (Reviewer)
- Oana Balalau, Assistant professor, INRIA, France (Examiner)
- Claire Gardent, Senior Researcher, CNRS, France (Examiner)
- Thomas Bonald, Full professor, Télécom Paris (LTCI), France (Thesis Director)
- Fragkiskos Malliaros, Full professor, CentraleSupélec, France (Thesis Co-supervisor)
Abstract
Reliable measurement of the factual knowledge stored in large language models (LLMs) presupposes reliable ground truth, yet existing benchmarks are typically noisy, narrow in domain, or incomplete. We introduce MUSART (MUlti-domain SAlient Relation Triples), a benchmark of factual triples automatically extracted from Wikidata by exploiting the categorical structure of Wikipedia’s « Good Articles » and a graph-based salience model based on Personalized PageRank.
MUSART probes the breadth of an LLM’s parametric knowledge across many domains while remaining cheap to reconstruct. To address knowledge-base incompleteness—a systematic source of false negatives in factuality evaluation—we further contribute an automated, source-grounded augmentation pipeline that mines additional valid facts from article text and stratifies them into empirically calibrated confidence tiers, turning MUSART into a tunable benchmark family. Evaluating a suite of open-weight LLMs, we show that (i) rigid exact-match scoring substantially underestimates factual recall, (ii) completeness correction raises measured precision while exposing a large long-tail recall gap, (iii) model rankings are unstable across evaluation settings, and (iv) factual competence is strongly domain-dependent. MUSART and its augmentation pipeline provide scalable, trustworthy ground truth for studying language models as knowledge bases.