--- title: "Dynamic Evaluation of Large Language Models by Meta Probing" description: "To address both issues, the authors propose Meta Probing Agents (MPA), presented at ICML 2024. 1 reference, cited by 29 papers here." authors: ["Kaijie Zhu", "Jindong Wang", "Qinlin Zhao", "Ruochen Xu", "Xing Xie"] year: 2024 venue: "arXiv:2402.14865" source: "https://arxiv.org/abs/2402.14865" doi: "10.48550/arxiv.2402.14865" arxiv: "2402.14865" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/zhu-2024-dynamic-evaluation-large-language-models/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhu-2024-dynamic-evaluation-large-language-models/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/) / Dynamic Evaluation of Large Language Models by Meta Probing Agents # Dynamic Evaluation of Large Language Models by Meta Probing Agents **Authors:** [Kaijie Zhu](https://md.research.elamur.ai/free/en/authors/kaijie-zhu/), [Jindong Wang](https://md.research.elamur.ai/free/en/authors/jindong-wang/), Qinlin Zhao, [Ruochen Xu](https://md.research.elamur.ai/free/en/authors/ruochen-xu/) and [Xing Xie](https://md.research.elamur.ai/free/en/authors/xing-xie/) · **Year:** 2024 · **Venue:** arXiv:2402.14865 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2402.14865) · [PDF](https://arxiv.org/pdf/2402.14865) · [DOI 10.48550/arxiv.2402.14865](https://doi.org/10.48550/arxiv.2402.14865) · [arXiv 2402.14865](https://arxiv.org/abs/2402.14865) · [OpenAlex](https://openalex.org/W4392181802) **Citations:** 1 citations (OpenAlex, 2026-10-14) · [29](https://md.research.elamur.ai/free/en/papers/zhu-2024-dynamic-evaluation-large-language-models/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Static public benchmarks can leak into LLM training data, so reported scores may overstate what models can actually do. Most of these benchmarks also give only a single aggregate score. Earlier dynamic protocols such as DyVal and NPHardEval rely on task-specific algorithms and mostly target reasoning. To address both issues, the authors propose Meta Probing Agents (MPA), presented at ICML 2024. MPA is grounded in psychometrics, which distinguishes three fundamental cognitive skills: comprehending language, solving problems and possessing domain knowledge. A probing agent, itself an LLM, rewrites existing test items according to principles linked to each ability. A judge agent then checks that the rewritten item still matches the original, and the two agents can iterate over several rounds. Principles can be mixed freely, which allows per-ability analysis. When MPA was applied to MMLU, GSM8K, BBH and ARC-C, most models scored noticeably lower than on the original sets, which points to possible contamination. The three abilities turned out to be strongly intercorrelated. MPA-generated data also proved useful for fine-tuning. Code is released in Microsoft's promptbench. ## Key points - **Models evaluated:** GPT-4-Turbo, GPT-3.5-Turbo, Gemini-Pro, Llama2-70b-chat, Yi-34b-chat and Mixtral-8x7b-Instruct. Llama2 7b and 13b variants were added for the scaling analysis. - **Example principles:** paraphrasing a question or its answer options, shuffling the options, inserting extra context, and adding a new option. Combining several principles lowered the scores of every model, and prompt engineering recovered only a little. - **Correlations between abilities:** Pearson, Spearman and Kendall coefficients were computed. The link between language understanding and problem solving was the strongest. - **Effect of model size:** larger models show tighter correlations between abilities, which the authors call an implicit "Matthew effect". Across the Llama2 sizes, all three abilities rise at a similar rate as models grow. - **Error analysis:** the authors manually examined 50 GSM8K cases where GPT-4-Turbo solved the original item but failed the transformed one. On MMLU, professional_law had the highest error rate, followed by moral_scenarios and professional_psychology. - **Data augmentation:** in a pilot study, GPT-3.5-Turbo was fine-tuned on MPA-transformed training splits of MMLU and ARC-C. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [University of Science and Technology of China](https://md.research.elamur.ai/free/en/universities/university-of-science-and-technology-of-china/) — University, Hefei, China · 108 works · 112,698 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works 🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works · [🔬 Interpretability](https://md.research.elamur.ai/free/en/topics/interpretability/) · 130 works ## Cite Cite as: Kaijie Zhu, Jindong Wang, Qinlin Zhao, Ruochen Xu and Xing Xie. 2024. Dynamic Evaluation of Large Language Models by Meta Probing Agents. arXiv:2402.14865. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{zhu2024dynamic, title = {{Dynamic Evaluation of Large Language Models by Meta Probing Agents}}, author = {Kaijie Zhu, Jindong Wang, Qinlin Zhao, Ruochen Xu and Xing Xie}, year = {2024}, doi = {10.48550/arxiv.2402.14865}, eprint = {2402.14865}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2402.14865} } ``` - [← previous Contextualization Distillation from Large Language Model for Knowledge Graph Completion](https://md.research.elamur.ai/free/en/papers/li-2024-contextualization-distillation-large-language-model/) - [next → Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languages](https://md.research.elamur.ai/free/en/papers/zhang-2024-enhancing-multilingual-capabilities-large-language/) - [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/) ## References 1 references, 0 of them with a card of ours. ↗ leads to the work. 1. 01-ai. Yi: A series of large language models. [https://github.com/01-ai/Yi](https://github.com/01-ai/Yi), 2024. *Figure 8. The performance of GPT-3.5-Turbo and GPT-3.5-Turbo-FT on original benchmarks and our probing benchmarks.* Bai, Y., Ying, J., Cao, Y., Lv, X., He, Y., Wang, X., Yu, J., Zeng, K., Xiao, Y., Lyu, H., et al. Ben… [↗](https://github.com/01-ai/Yi) *Updated 31 October 2026*