--- title: "PIRA — paper by Xue (2026) — summary and ideas" description: "PIRA targets two weaknesses of discriminative reward models used in RLHF. 1 reference." authors: ["Yongfu Xue"] year: 2026 venue: "Findings of the Association for Computational Linguistics: EACL 2026" source: "https://aclanthology.org/2026.findings-eacl.117/" doi: "10.18653/v1/2026.findings-eacl.117" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/xue-2026-pira-preference-oriented-instruction-tuned/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xue-2026-pira-preference-oriented-instruction-tuned/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/45/) / PIRA: Preference-Oriented Instruction-Tuned Reward Models with Dual Aggregation # PIRA: Preference-Oriented Instruction-Tuned Reward Models with Dual Aggregation **Authors:** Yongfu Xue · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: EACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.findings-eacl.117/) · [PDF](https://aclanthology.org/2026.findings-eacl.117.pdf) · [DOI 10.18653/v1/2026.findings-eacl.117](https://doi.org/10.18653/v1/2026.findings-eacl.117) · [OpenAlex](https://openalex.org/W7140085809) **Citations:** 0 citations (OpenAlex, 2026-10-14) ## Summary PIRA targets two weaknesses of discriminative reward models used in RLHF. First, they need large amounts of labeled data because they feed a plain question–answer concatenation and never tell the LLM that the task is scoring. Second, policies tend to exploit them (reward overoptimization). PIRA addresses both with three strategies: - **Instruction framing.** Each input is wrapped in a preference-evaluation instruction drawn from a set that an LLM generated and humans reviewed. - **Instruction averaging.** At inference, rewards are averaged across several such instructions. - **Value-head dropout averaging.** Many dropout passes, at varying rates, are run through the lightweight value head only, on top of a single deterministic backbone pass. Training uses the Bradley–Terry loss with LoRA and separate learning rates for the backbone and the head. Across four backbones and six public preference datasets, PIRA mostly beats a plain baseline, Thomas and WARM. In PPO experiments, it keeps KL divergence bounded and lets the gold reward rise steadily instead of collapsing. The approach therefore gives cheaper, more stable reward models that are harder to hack. ## Key points - **Setup:** Backbones were Mistral-7B-v0.1, LLaMA3-8B, Qwen2.5-1.5B and Qwen2.5-7B. Datasets were HH, HH-cleaned, SHP, Alpaca-farm, Oasst and UltraFeedback. Inference used K=6 instructions and M=12 dropout passes, with rates sampled from Uniform(0.1, 0.4). - **Average accuracy gains over the baseline:** - Mistral-7B: 65.7 → 70.1 - LLaMA3-8B: 65.0 → 69.8 - Qwen2.5-7B: 63.5 → 69.8 - Qwen2.5-1.5B: 62.1 → 66.1, where WARM* is marginally higher at 66.2 - **Ablation (LLaMA3-8B, HH-cleaned):** Instruction framing alone lifts accuracy from 64.3 to 73.0. Adding both averaging steps reaches 75.5 and cuts the standard deviation from 2.1 to 0.7. - **Cost:** Value-head averaging adds only about 7% latency at M=12, while K=6 instructions multiply inference cost roughly sixfold. - **Reward-hacking test:** PPO used Qwen2.5-1.5B as policy and reward model, with Qwen2.5-7B as the gold judge, on Alpaca-farm. PIRA suppressed reward inflation compared with the baseline and Thomas. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works ## Cite Cite as: Yongfu Xue. 2026. PIRA: Preference-Oriented Instruction-Tuned Reward Models with Dual Aggregation. Findings of the Association for Computational Linguistics: EACL 2026. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{xue2026pira, title = {{PIRA: Preference-Oriented Instruction-Tuned Reward Models with Dual Aggregation}}, author = {Yongfu Xue}, booktitle = {Findings of the Association for Computational Linguistics: EACL 2026}, year = {2026}, doi = {10.18653/v1/2026.findings-eacl.117}, url = {https://doi.org/10.18653/v1/2026.findings-eacl.117} } ``` - [← previous Pioneering Bot Detection on Polish Reddit at the Comment Level](https://md.research.elamur.ai/free/en/papers/matyjaszek-2026-pioneering-bot-detection-polish-reddit/) - [next → Pixel Phantoms at SemEval-2026 Task 13: Exploring Classical and Neural Approaches for AI-Generated Code Detection](https://md.research.elamur.ai/free/en/papers/s-2026-pixel-phantoms-semeval-2026-task-2026-semeval-1-256/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/45/) ## References 1 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan D Chang, and Prithviraj Ammanabrolu. 2024. Critique-out-loud reward models. arXiv preprint arXiv:2408.11791. +0 +2 +3 +5 +7 *Table 4: Impact of Value-Head Stochastic Averaging.* [find](https://scholar.google.com/scholar?q=Zachary+Ankner%2C+Mansheej+Paul%2C+Brandon+Cui%2C+Jonathan+D+Chang%2C+and+Prithviraj+Ammanabrolu%2E+2024%2E+Critique%2Dout%2Dloud+reward+models%2E+arXiv+preprint+arXiv%3A2408%2E11791%2E+%2B0+%2B2+%2B3+%2B5+%2B7+%2ATable+4%3A+Impact+of+Val) *Updated 2 November 2026*