---
title: "The Trickle-down Impact of Reward (In-)consistency on RLHF"
description: "The authors propose two fixes that add no training cost: ConvexDA, used during RM training, and… 79 references, cited by 10 papers here."
authors: ["Lingfeng Shen", "Sihao Chen", "Linfeng Song", "Lifeng Jin", "Baolin Peng", "Haitao Mi", "Daniel Khashabi", "Dong Yu"]
year: 2023
venue: "arXiv:2309.16155"
source: "https://arxiv.org/abs/2309.16155"
doi: "10.48550/arxiv.2309.16155"
arxiv: "2309.16155"
licence: "CC BY-SA 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-sa/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shen-2023-trickle-down-impact-reward-consistency/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shen-2023-trickle-down-impact-reward-consistency/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/13/) / The Trickle-down Impact of Reward (In-)consistency on RLHF

# The Trickle-down Impact of Reward (In-)consistency on RLHF

**Authors:** [Lingfeng Shen](https://md.research.elamur.ai/free/en/authors/lingfeng-shen/), [Sihao Chen](https://md.research.elamur.ai/free/en/authors/sihao-chen/), [Linfeng Song](https://md.research.elamur.ai/free/en/authors/linfeng-song/), [Lifeng Jin](https://md.research.elamur.ai/free/en/authors/lifeng-jin/), [Baolin Peng](https://md.research.elamur.ai/free/en/authors/baolin-peng/), [Haitao Mi](https://md.research.elamur.ai/free/en/authors/haitao-mi/), [Daniel Khashabi](https://md.research.elamur.ai/free/en/authors/daniel-khashabi/) and [Dong Yu](https://md.research.elamur.ai/free/en/authors/dong-yu/) · **Year:** 2023 · **Venue:** arXiv:2309.16155 · **Licence:** [CC BY-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-sa/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2309.16155) · [PDF](https://arxiv.org/pdf/2309.16155) · [DOI 10.48550/arxiv.2309.16155](https://doi.org/10.48550/arxiv.2309.16155) · [arXiv 2309.16155](https://arxiv.org/abs/2309.16155) · [OpenAlex](https://openalex.org/W6884966045)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [10](https://md.research.elamur.ai/free/en/papers/shen-2023-trickle-down-impact-reward-consistency/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Reward models (RMs) guide RLHF training, but few studies check whether an RM reacts correctly when a prompt changes slightly in wording and substantially in meaning. The authors call this property reward (in-)consistency. They ask how to measure it, how well current RMs handle it, and whether it affects the final chatbot. Their benchmarking recipe, ContrastInstructions, is built automatically from human preference data. It pairs nearly identical instructions that need different answers, then checks whether the RM scores each instruction with its own answer above the swapped combinations. RMs trained with the usual pairwise ranking loss, such as a LLaMa-7B-based RM, score close to chance. Humans pick correctly about 80% of the time. The authors propose two fixes that add no training cost: ConvexDA, used during RM training, and RewardFusion, used at inference. Both improve consistency, but a wide gap to humans remains. PPO policies trained against more consistent RMs produce more useful answers. This suggests that RM flaws carry over into the final model and that consistency can serve as an intrinsic check on RMs.

## Key points
- The paper defines two metrics. Response consistency (C_res) tests whether the RM prefers the correct answer for a given instruction. Instruction consistency (C_ins) tests whether it prefers the correct instruction for a given answer. Standard RMs are expected to do better on C_res, since their training objective only ranks responses.
- ContrastInstructions is built from four datasets: StackExchange (QA), WMT (machine translation), RealSumm (summarization) and Twitter (paraphrase generation).
- Instruction pairs are selected with SimCSE embeddings. Only pairs with cosine similarity between 0.75 and 0.9 are kept, so they are similar but not equivalent.
- Standard ranking-trained RMs perform near random on the benchmark, while humans reach roughly 80%. ConvexDA and RewardFusion narrow this gap only partially.
- The benchmark datasets and code are released at github.com/shadowkiller33/Contrast-Instruction (arXiv:2309.16155, Johns Hopkins, UPenn and Tencent AI Lab).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Pennsylvania](https://md.research.elamur.ai/free/en/universities/university-of-pennsylvania/) — University, Philadelphia, United States · 233 works · 65,583 citations
- [Johns Hopkins University](https://md.research.elamur.ai/free/en/universities/johns-hopkins-university/) — University, Baltimore, United States · 247 works · 106,235 citations
- [Tencent](https://md.research.elamur.ai/free/en/companies/tencent/) — Company, Shenzhen, China · 163 works · 21,032 citations

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works

## Cite

Cite as: Lingfeng Shen, Sihao Chen, Linfeng Song, Lifeng Jin, Baolin Peng, Haitao Mi, Daniel Khashabi and Dong Yu. 2023. The Trickle-down Impact of Reward (In-)consistency on RLHF. arXiv:2309.16155.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{shen2023trickle,
  title = {{The Trickle-down Impact of Reward (In-)consistency on RLHF}},
  author = {Lingfeng Shen, Sihao Chen, Linfeng Song, Lifeng Jin, Baolin Peng, Haitao Mi, Daniel Khashabi and Dong Yu},
  year = {2023},
  doi = {10.48550/arxiv.2309.16155},
  eprint = {2309.16155},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2309.16155}
}
```

- [← previous TCE at Qur'an QA 2023 Shared Task: Low Resource Enhanced Transformer-based Ensemble Approach for Qur'anic QA](https://md.research.elamur.ai/free/en/papers/elkomy-2023-tce-qur-qa-2023-shared/)
- [next → Towards Generative Event Factuality Prediction](https://md.research.elamur.ai/free/en/papers/murzaku-2023-generative-event-factuality-prediction/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/13/)

## References

79 references, 39 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Sharat Agarwal, Himanshu Arora, Saket Anand, and Chetan Arora. Contextual diversity for active learning. In The European Conference on Computer Vision (ECCV), pp. 137–153. Springer, 2020. URL https://arxiv.org/abs/2008.05723. — [arXiv:2008.05723](https://arxiv.org/abs/2008.05723) [↗](https://arxiv.org/abs/2008.05723)
2. Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. [Synthetic QA corpora generation with roundtrip consistency](https://md.research.elamur.ai/free/en/papers/alberti-2019-synthetic-qa-corpora-generation-roundtrip/). In Anna Korhonen, David R. Traum, and Lluı́s Màrquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Flore…
3. Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani B Srivastava, and Kai-Wei Chang. [Generating natural language adversarial examples](https://md.research.elamur.ai/free/en/papers/alzantot-2018-generating-natural-language-adversarial-examples/). In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018. URL https://arxiv.org/abs/1804. 07998. [↗](https://arxiv.org/abs/1804)
4. Akari Asai and Hannaneh Hajishirzi. [Logic-guided data augmentation and regularization for consistent question answering](https://md.research.elamur.ai/free/en/papers/asai-2020-logic-guided-data-augmentation-regularization/). In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Onl…
5. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021. URL https://arxiv.org/abs/ 2112.00861. [↗](https://arxiv.org/abs/)
6. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022a. URL https://arxiv.org/abs/… [↗](https://arxiv.org/abs/…)
7. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. [Constitutional ai: Harmlessness from ai feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/). arXiv preprint arXiv:2212.08073, 2022b. URL https://arxiv.org/abs/2212.08073. [↗](https://arxiv.org/abs/2212.08073)
8. Edward Beeching, Younes Belkada, Kashif Rasul, Lewis Tunstall, Leandro von Werra, Nazneen Rajani, and Nathan Lambert. Stackllama: An rl fine-tuned llama model for stack exchange question and answering, 2023. URL [https://huggingface.co/blog/stackllama](https://huggingface.co/blog/stackllama). [↗](https://huggingface.co/blog/stackllama)
9. Yonatan Belinkov and Yonatan Bisk. Synthetic and natural noise both break neural machine translation. In International Conference on Learning Representations (ICLR), 2018. URL https://arxiv.org/abs/1711.02173. [↗](https://arxiv.org/abs/1711.02173)
10. Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu, and Graham Neubig. [Re-evaluating evaluation in text summarization](https://md.research.elamur.ai/free/en/papers/bhandari-2020-re-evaluating-evaluation-text-summarization/). In Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9347–9359, 2020. URL https://arxiv.org/abs/ 2010.07100. [↗](https://arxiv.org/abs/)
11. Oana-Maria Camburu, Brendan Shillingford, Pasquale Minervini, Thomas Lukasiewicz, and Phil Blunsom. Make up your mind! adversarial generation of inconsistent natural language explanations. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Mee… [find](https://scholar.google.com/scholar?q=Oana%2DMaria+Camburu%2C+Brendan+Shillingford%2C+Pasquale+Minervini%2C+Thomas+Lukasiewicz%2C+and+Phil+Blunsom%2E+Make+up+your+mind%21+adversarial+generation+of+inconsistent+natural+language+explanations%2E+In+Dan+Jura)
12. Sihao Chen, Fan Zhang, Kazoo Sone, and Dan Roth. [Improving faithfulness in abstractive summarization with contrast candidate generation and selection](https://md.research.elamur.ai/free/en/papers/chen-2021-improving-faithfulness-abstractive-summarization-contrast/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp…
13. Xiaoyi Chen, Ahmed Salem, Michael Backes, Shiqing Ma, and Yang Zhang. Badnl: Backdoor attacks against NLP models. In ICML Workshop on Adversarial Machine Learning, 2021b. URL https://arxiv.org/abs/2006.01043. — [arXiv:2006.01043](https://arxiv.org/abs/2006.01043) [↗](https://arxiv.org/abs/2006.01043)
14. Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526, 2017. [find](https://scholar.google.com/scholar?q=Xinyun+Chen%2C+Chang+Liu%2C+Bo+Li%2C+Kimberly+Lu%2C+and+Dawn+Song%2E+Targeted+backdoor+attacks+on+deep+learning+systems+using+data+poisoning%2E+arXiv+preprint+arXiv%3A1712%2E05526%2C+2017%2E)
15. Yutian Chen, Max Welling, and Alex Smola. Super-samples from kernel herding. In Conference on Uncertainty in Artificial Intelligence (UAI), pp. 109–116, 2010. URL [https://arxiv.org/](https://arxiv.org/) abs/1203.3472. [↗](https://arxiv.org/)
16. Jiazhu Dai, Chuanshuai Chen, and Yufeng Li. A backdoor attack against lstm-based text classification systems. IEEE Access, 7:138872–138878, 2019. URL https://arxiv.org/abs/ 1905.12457. [↗](https://arxiv.org/abs/)
17. Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. [Raft: reward ranked finetuning for generative foundation model alignment](https://md.research.elamur.ai/free/en/papers/dong-2023-raft-reward-ranked-finetuning-generative/). arXiv preprint arXiv:2304.06767, 2023. URL https://arxiv.org/abs/2304.06767. [↗](https://arxiv.org/abs/2304.06767)
18. Xinya Du, Bhavana Dalvi Mishra, Niket Tandon, Antoine Bosselut, Wen-tau Yih, Peter Clark, and Claire Cardie. Be consistent! improving procedural text comprehension using label consistency. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North Ame… [find](https://scholar.google.com/scholar?q=Xinya+Du%2C+Bhavana+Dalvi+Mishra%2C+Niket+Tandon%2C+Antoine+Bosselut%2C+Wen%2Dtau+Yih%2C+Peter+Clark%2C+and+Claire+Cardie%2E+Be+consistent%21+improving+procedural+text+comprehension+using+label+consistency%2E+In+Jill+Bur)
19. Steffen Eger, Gözde Gül Şahin, Andreas Rücklé, Ji-Ung Lee, Claudia Schulz, Mohsen Mesgar, Krishnkant Swarnkar, Edwin Simpson, and Iryna Gurevych. [Text processing like humans do: Visually attacking and shielding nlp systems](https://md.research.elamur.ai/free/en/papers/eger-2019-text-processing-like-humans-do/). In Conference of the North American Chapter of the Association for Comp…
20. Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard H. Hovy, Hinrich Schütze, and Yoav Goldberg. Erratum: Measuring and improving consistency in pretrained language models. Trans. Assoc. Comput. Linguistics, 9:1407, 2021. doi: 10.1162/tacl\\ x\\ 00455. URL https://doi.org/10.11… [find](https://scholar.google.com/scholar?q=Yanai+Elazar%2C+Nora+Kassner%2C+Shauli+Ravfogel%2C+Abhilasha+Ravichander%2C+Eduard+H%2E+Hovy%2C+Hinrich+Schu%CC%88tze%2C+and+Yoav+Goldberg%2E+Erratum%3A+Measuring+and+improving+consistency+in+pretrained+language+models%2E+Tra)
21. Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. [Ppdb: The paraphrase database](https://md.research.elamur.ai/free/en/papers/ganitkevitch-2013-ppdb-paraphrase-database/). In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 758–764, 2013. URL https://aclanthology.org/N13-1092… [↗](https://aclanthology.org/N13-1092…)
22. Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. [Black-box generation of adversarial text sequences to evade deep learning classifiers](https://md.research.elamur.ai/free/en/papers/gao-2018-black-box-generation-adversarial-text/). In IEEE Security and Privacy Workshops (SPW), 2018. URL https://arxiv.org/abs/1801.04354. [↗](https://arxiv.org/abs/1801.04354)
23. Leo Gao, John Schulman, and Jacob Hilton. [Scaling laws for reward model overoptimization](https://md.research.elamur.ai/free/en/papers/gao-2022-scaling-laws-reward-model-overoptimization/). arXiv preprint arXiv:2210.10760, 2022. URL https://arxiv.org/abs/2210.10760. [↗](https://arxiv.org/abs/2210.10760)
24. Tianyu Gao, Xingcheng Yao, and Danqi Chen. [Simcse: Simple contrastive learning of sentence embeddings](https://md.research.elamur.ai/free/en/papers/gao-2021-simcse-simple-contrastive-learning-sentence/). arXiv preprint arXiv:2104.08821, 2021. URL https://arxiv.org/abs/ 2104.08821. [↗](https://arxiv.org/abs/)
25. Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, et al. Evaluating models’ local decision boundaries via contrast sets. In Conference on Empirical Methods in Natural Language Processing (EMNLP) - Findin… — [Evaluating Models' Local Decision Boundaries via Contrast Sets](https://md.research.elamur.ai/free/en/papers/gardner-2020-evaluating-models-local-decision-boundaries/)
26. Amelia Glaese, Nat McAleese, Maja Trbacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. [Improving alignment of dialogue agents via targeted human judgements](https://md.research.elamur.ai/free/en/papers/glaese-2022-improving-alignment-dialogue-agents-targeted/). arXiv preprint arXiv:2209.14375, 2022. URL https://arxiv.org/abs/2209.1437… [↗](https://arxiv.org/abs/2209.1437…)
27. Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, Nahyeon Ryu, and Marc Dymetman. [Aligning language models with preferences through f-divergence minimization](https://md.research.elamur.ai/free/en/papers/go-2023-aligning-language-models-preferences-through/). arXiv preprint arXiv:2302.08215, 2023. URL https://arxiv.org/abs/2302.08215. [↗](https://arxiv.org/abs/2302.08215)
28. Karan Goel, Nazneen Fatema Rajani, Jesse Vig, Zachary Taschdjian, Mohit Bansal, and Christopher Ré. Robustness gym: Unifying the nlp evaluation landscape. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologie… [find](https://scholar.google.com/scholar?q=Karan+Goel%2C+Nazneen+Fatema+Rajani%2C+Jesse+Vig%2C+Zachary+Taschdjian%2C+Mohit+Bansal%2C+and+Christopher+Re%CC%81%2E+Robustness+gym%3A+Unifying+the+nlp+evaluation+landscape%2E+In+Proceedings+of+the+2021+Conference+of+the)
29. Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. [Badnets: Identifying vulnerabilities in the machine learning model supply chain](https://md.research.elamur.ai/free/en/papers/gu-2017-badnets-identifying-vulnerabilities-machine-learning/). arXiv preprint arXiv:1708.06733, 2017. URL https://arxiv.org/abs/1708.06733. [↗](https://arxiv.org/abs/1708.06733)
30. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321–1330. PMLR, 2017. URL [https://arxiv.org/pdf/1706.04599.pdf](https://arxiv.org/pdf/1706.04599.pdf). [↗](https://arxiv.org/pdf/1706.04599.pdf)
31. Xuanli He, Lingjuan Lyu, Lichao Sun, and Qiongkai Xu. Model extraction and adversarial transferability, your bert is vulnerable! In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pp. 2006–2012, 2021. URL https://arxiv.org/abs/2103.10013. [↗](https://arxiv.org/abs/2103.10013)
32. Donald Joseph Hejna III and Dorsa Sadigh. Few-shot preference learning for human-in-the-loop RL. In Conference on Robot Learning (CoRL), pp. 2014–2025, 2023. URL https://arxiv.org/abs/2212.03363. — [arXiv:2212.03363](https://arxiv.org/abs/2212.03363) [↗](https://arxiv.org/abs/2212.03363)
33. Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In International Conference on Learning Representations (ICLR), 2021. URL https://arxiv.org/abs/2106.09685. [↗](https://arxiv.org/abs/2106.09685)
34. Po-Sen Huang, Robert Stanforth, Johannes Welbl, Chris Dyer, Dani Yogatama, Sven Gowal, Krishnamurthy Dvijotham, and Pushmeet Kohli. [Achieving verified robustness to symbol substitutions via interval bound propagation](https://md.research.elamur.ai/free/en/papers/huang-2019-achieving-verified-robustness-symbol-substitutions/). In Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 408…
35. Robin Jia and Percy Liang. [Adversarial examples for evaluating reading comprehension systems](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/). In EMNLP, 2017. URL https://arxiv.org/abs/1707.07328. [↗](https://arxiv.org/abs/1707.07328)
36. Robin Jia, Aditi Raghunathan, Kerem Göksel, and Percy Liang. [Certified robustness to adversarial word substitutions](https://md.research.elamur.ai/free/en/papers/jia-2019-certified-robustness-adversarial-word-substitutions/). In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019. URL https://arxiv.org/abs/1909.00986. [↗](https://arxiv.org/abs/1909.00986)
37. Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. [Is bert really robust? a strong baseline for natural language attack on text classification and entailment](https://md.research.elamur.ai/free/en/papers/jin-2019-bert-really-robust-strong-baseline/). In Conference on Artificial Intelligence (AAAI), 2020. URL https://arxiv.org/abs/1907.11932. [↗](https://arxiv.org/abs/1907.11932)
38. Jeff Johnson, Matthijs Douze, and Hervé Jégou. [Billion-scale similarity search with GPUs](https://md.research.elamur.ai/free/en/papers/johnson-2017-billion-scale-similarity-search-gpus/). IEEE Transactions on Big Data, 7(3):535–547, 2019. URL https://arxiv.org/abs/1702. 08734. [↗](https://arxiv.org/abs/1702)
39. Nora Kassner, Oyvind Tafjord, Hinrich Schutze, and Peter Clark. Enriching a model’s notion of belief using a persistent memory. arXiv preprint arXiv:2104.08401, 2021. URL https://arxiv.org/abs/2104.08401. — [arXiv:2104.08401](https://arxiv.org/abs/2104.08401) [↗](https://arxiv.org/abs/2104.08401)
40. Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. [Pretraining language models with human preferences](https://md.research.elamur.ai/free/en/papers/korbak-2023-pretraining-language-models-human-preferences/). arXiv preprint arXiv:2302.08582, 2023. URL [https://openreview.net/pdf?](https://openreview.net/pdf?) id=AT8Iw8KOeC. [↗](https://openreview.net/pdf?)
41. Thai Le, Noseong Park, and Dongwon Lee. A sweet rabbit hole by darcy: Using honeypots to detect universal trigger’s adversarial attacks. In Annual Meeting of the Association for Computational Linguistics (ACL), pp. 3831–3844. Association for Computational Linguistics (ACL), 2021. URL [https://arxiv.o…](https://arxiv.o…) [↗](https://arxiv.o…)
42. Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267, 2023. URL [https://arxiv.org/pdf/](https://arxiv.org/pdf/) 2309.00267.pdf. [↗](https://arxiv.org/pdf/)
43. Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. [Quark: controllable text generation with reinforced unlearning](https://md.research.elamur.ai/free/en/papers/lu-2022-quark-controllable-text-generation-reinforced/). Advances in neural information processing systems, 35:27591–27609, 2022. URL https://arxiv.org/abs/2205.13636. [↗](https://arxiv.org/abs/2205.13636)
44. Edward Ma. NLP augmentation, 2019. URL [https://github.com/makcedward/nlpaug](https://github.com/makcedward/nlpaug). [↗](https://github.com/makcedward/nlpaug)
45. Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. [Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges](https://md.research.elamur.ai/free/en/papers/ma-2019-results-wmt19-metrics-shared-task/). In Conference on Machine Translation (WMT), 2019. URL [https://www.aclweb.org/anthology/W19-5302](https://www.aclweb.org/anthology/W19-5302). [↗](https://www.aclweb.org/anthology/W19-5302)
46. Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, and Dan Roth. [Expertqa: Expert-curated questions and attributed answers](https://md.research.elamur.ai/free/en/papers/malaviya-2024-expertqa-expert-curated-questions-attributed/). arXiv preprint arXiv:2309.07852, 2023.
47. David Manheim and Scott Garrabrant. Categorizing variants of goodhart’s law. arXiv preprint arXiv:1803.04585, 2018. URL https://arxiv.org/abs/1803.04585. — [arXiv:1803.04585](https://arxiv.org/abs/1803.04585) [↗](https://arxiv.org/abs/1803.04585)
48. George Miller. [Wordnet: a lexical database for english](https://md.research.elamur.ai/free/en/papers/miller-1994-wordnet/). Communications of the ACM, 38(11):39–41, 1995. URL [https://dl.acm.org/doi/10.1145/219717.219748](https://dl.acm.org/doi/10.1145/219717.219748). [↗](https://dl.acm.org/doi/10.1145/219717.219748)
49. Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-Task Generalization via Natural Language Crowdsourcing Instructions. In Annual Meeting of the Association for Computational Linguistics (ACL), 2022. URL https://arxiv.org/abs/2104.08773. [↗](https://arxiv.org/abs/2104.08773)
50. Eric Mitchell, Joseph J. Noh, Siyan Li, William S. Armstrong, Ananth Agarwal, Patrick Liu, Chelsea Finn, and Christopher D. Manning. Enhancing self-consistency and performance of pre-trained language models through natural language inference. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.)… [find](https://scholar.google.com/scholar?q=Eric+Mitchell%2C+Joseph+J%2E+Noh%2C+Siyan+Li%2C+William+S%2E+Armstrong%2C+Ananth+Agarwal%2C+Patrick+Liu%2C+Chelsea+Finn%2C+and+Christopher+D%2E+Manning%2E+Enhancing+self%2Dconsistency+and+performance+of+pre%2Dtrained+language+)
51. Maximilian Mozes, Pontus Stenetorp, Bennett Kleinberg, and Lewis Griffin. Frequency-guided word substitutions for detecting textual adversarial examples. In Conference of the European Chapter of the Association for Computational Linguistics (EACL), pp. 171–186, 2021. URL https://arxiv.org/abs/2004.0… [↗](https://arxiv.org/abs/2004.0…)
52. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. [WebGPT: Browser-assisted question-answering with human feedback](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/). arXiv preprint arXiv:2112.09332, 2021. URL https://arxiv.org/abs/2112.09332… [↗](https://arxiv.org/abs/2112.09332…)
53. OpenAI. [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/), 2023. URL https://arxiv.org/abs/2303.08774. [↗](https://arxiv.org/abs/2303.08774)
54. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. [Training Language Models to Follow Instructions with Human Feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). In Advances in Neural Information Processing Systems (NeurIPS), 2022. URL https://…
55. Fanchao Qi, Yangyi Chen, Mukai Li, Yuan Yao, Zhiyuan Liu, and Maosong Sun. [Onion: A simple and effective defense against textual backdoor attacks](https://md.research.elamur.ai/free/en/papers/qi-2021-onion-simple-effective-defense-against/). arXiv preprint arXiv:2011.10369, 2020. URL https://aclanthology.org/2021.emnlp-main.752/. [↗](https://aclanthology.org/2021.emnlp-main.752/)
56. Fanchao Qi, Yangyi Chen, Xurui Zhang, Mukai Li, Zhiyuan Liu, and Maosong Sun. [Mind the style of text! adversarial and backdoor attacks based on text style transfer](https://md.research.elamur.ai/free/en/papers/qi-2021-mind-style-text-adversarial-backdoor/). In Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4569–4580, 2021a. URL https://arxiv.org/abs/2110.07139. [↗](https://arxiv.org/abs/2110.07139)
57. Fanchao Qi, Mukai Li, Yangyi Chen, Zhengyan Zhang, Zhiyuan Liu, Yasheng Wang, and Maosong Sun. [Hidden killer: Invisible textual backdoor attacks with syntactic trigger](https://md.research.elamur.ai/free/en/papers/qi-2021-hidden-killer-invisible-textual-backdoor/). arXiv preprint arXiv:2105.12400, 2021b. URL https://arxiv.org/abs/2105.12400. [↗](https://arxiv.org/abs/2105.12400)
58. Shashank Rajput, Zhili Feng, Zachary Charles, Po-Ling Loh, and Dimitris Papailiopoulos. Does data augmentation lead to positive margin? In International Conference on Learning Representations (ICLR), pp. 5321–5330, 2019. URL https://arxiv.org/abs/1905.03177. — [arXiv:1905.03177](https://arxiv.org/abs/1905.03177) [↗](https://arxiv.org/abs/1905.03177)
59. Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. [Generating natural language adversarial examples through probability weighted word saliency](https://md.research.elamur.ai/free/en/papers/ren-2019-generating-natural-language-adversarial-examples/). In Annual Meeting of the Association for Computational Linguistics (ACL), 2019. URL https://aclanthology.org/ P19-1103/. [↗](https://aclanthology.org/)
60. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Semantically equivalent adversarial rules for debugging nlp models. In Annual Meeting of the Association for Computational Linguistics (ACL), 2018. URL https://aclanthology.org/P18-1079/. [↗](https://aclanthology.org/P18-1079/)
61. Marco Túlio Ribeiro, Carlos Guestrin, and Sameer Singh. Are red roses red? evaluating consistency of question-answering models. In Anna Korhonen, David R. Traum, and Lluı́s Màrquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Ital… [find](https://scholar.google.com/scholar?q=Marco+Tu%CC%81lio+Ribeiro%2C+Carlos+Guestrin%2C+and+Sameer+Singh%2E+Are+red+roses+red%3F+evaluating+consistency+of+question%2Danswering+models%2E+In+Anna+Korhonen%2C+David+R%2E+Traum%2C+and+Llu%C4%B1%CC%81s+Ma%CC%80rquez+%28eds%2E%29%2C+Proceedin)
62. Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy: Behavioral testing of nlp models with checklist. In Annual Meeting of the Association for Computational Linguistics (ACL), pp. 4902–4912, 2020. URL https://aclanthology.org/ 2020.acl-main.442/. [↗](https://aclanthology.org/)
63. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. URL https://arxiv.org/abs/1707.06347. [↗](https://arxiv.org/abs/1707.06347)
64. Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations (ICLR), 2018. URL https://arxiv.org/abs/1708.00489. [↗](https://arxiv.org/abs/1708.00489)
65. Lingfeng Shen, Haiyun Jiang, Lemao Liu, and Shuming Shi. Rethink stealthy backdoor attacks in natural language processing. arXiv preprint arXiv:2201.02993, 2022a. URL https://arxiv.org/abs/2201.02993. — [arXiv:2201.02993](https://arxiv.org/abs/2201.02993) [↗](https://arxiv.org/abs/2201.02993)
66. Lingfeng Shen, Shoushan Li, and Ying Chen. KATG: keyword-bias-aware adversarial text generation for text classification. In Conference on Artificial Intelligence (AAAI), 2022b. URL [https://ojs.aaai.org/index.php/AAAI/article/view/21380](https://ojs.aaai.org/index.php/AAAI/article/view/21380). [↗](https://ojs.aaai.org/index.php/AAAI/article/view/21380)
67. Lingfeng Shen, Lemao Liu, Haiyun Jiang, and Shuming Shi. On the evaluation metrics for paraphrase generation. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 3178–3190, 2022c. URL https://arxiv.org/abs/2202.08479. [↗](https://arxiv.org/abs/2202.08479)
68. Lingfeng Shen, Ze Zhang, Haiyun Jiang, and Ying Chen. Textshield: Beyond successfully detecting adversarial sentences in text classification. In International Conference on Learning Representations (ICLR), 2023. URL https://arxiv.org/abs/2302.02023. — [arXiv:2302.02023](https://arxiv.org/abs/2302.02023) [↗](https://arxiv.org/abs/2302.02023)
69. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. [LLaMA: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023. URL https://arxiv.org/abs/2302.13971. [↗](https://arxiv.org/abs/2302.13971)
70. Boxin Wang, Chejian Xu, Xiangyu Liu, Yu Cheng, and Bo Li. Semattack: Natural textual attacks via different semantic spaces. arXiv preprint arXiv:2205.01287, 2022a. URL https://arxiv.org/abs/2205.01287. [↗](https://arxiv.org/abs/2205.01287)
71. Xiaosen Wang, Jin Hao, Yichen Yang, and Kun He. Natural language adversarial defense through synonym encoding. In Conference on Uncertainty in Artificial Intelligence (UAI), 2021. URL https://arxiv.org/abs/1909.06723. — [arXiv:1909.06723](https://arxiv.org/abs/1909.06723) [↗](https://arxiv.org/abs/1909.06723)
72. Xuezhi Wang, Haohan Wang, and Diyi Yang. [Measure and improve robustness in NLP models: A survey](https://md.research.elamur.ai/free/en/papers/wang-2022-measure-improve-robustness-nlp-models/). In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pp. 4569–4586, 2022b. URL https://arxiv.org/abs/2112.08313. [↗](https://arxiv.org/abs/2112.08313)
73. Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. [Fine-grained human feedback gives better rewards for language model training](https://md.research.elamur.ai/free/en/papers/wu-2023-fine-grained-human-feedback-gives/). In Advances in Neural Information Processing Systems, 2023. URL [https://arxiv.org/pdf/…](https://arxiv.org/pdf/…) [↗](https://arxiv.org/pdf/…)
74. Xinze Zhang, Junzhe Zhang, Zhenhua Chen, and Kun He. Crafting adversarial examples for neural machine translation. In Annual Meeting of the Association for Computational Linguistics (ACL), pp. 1967–1977, 2021. URL https://aclanthology.org/2021.acl-long. 153.pdf. [↗](https://aclanthology.org/2021.acl-long)
75. Yuan Zhang, Jason Baldridge, and Luheng He. [Paws: Paraphrase adversaries from word scrambling](https://md.research.elamur.ai/free/en/papers/zhang-2019-paws-paraphrase-adversaries-word-scrambling/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 1298–1308, 2019. URL…
76. Jake Zhao and Kyunghyun Cho. Retrieval-augmented convolutional neural networks against adversarial examples. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 11563– 11571. Computer Vision Foundation / IEEE, 2019. doi: 10.1109/CVPR.2… [find](https://scholar.google.com/scholar?q=Jake+Zhao+and+Kyunghyun+Cho%2E+Retrieval%2Daugmented+convolutional+neural+networks+against+adversarial+examples%2E+In+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+CVPR+2019%2C+Long+Beach%2C+CA%2C+U)
77. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). arXiv preprint arXiv:2306.05685, 2023. URL https://arxiv.org/abs/ 2306.05685. [↗](https://arxiv.org/abs/)
78. Banghua Zhu, Jiantao Jiao, and Michael Jordan. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. In ICLR Workshop on Mathematical and Empirical Understanding of Foundation Models, 2023. URL https://arxiv.org/abs/2301. 11270. [↗](https://arxiv.org/abs/2301)
79. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. URL https://arxiv.org/abs/1909.08593. [↗](https://arxiv.org/abs/1909.08593)

*Updated 30 October 2026*