---
title: "RAT — paper by Wang et al. (2024) — summary and ideas"
description: "The authors propose Retrieval Augmented Thoughts (RAT), a zero-shot prompting scheme that… 68 references, cited by 36 papers here."
authors: ["Zihao Wang", "Anji Liu", "Haowei Lin", "Jiaqi Li", "Xiaojian Ma", "Yitao Liang"]
year: 2024
venue: "arXiv:2403.05313"
source: "https://arxiv.org/abs/2403.05313"
doi: "10.48550/arxiv.2403.05313"
arxiv: "2403.05313"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2024-rat-retrieval-augmented-thoughts-elicit/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2024-rat-retrieval-augmented-thoughts-elicit/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/) / RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Horizon Generation

# RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Horizon Generation

**Authors:** [Zihao Wang](https://md.research.elamur.ai/free/en/authors/zihao-wang/), [Anji Liu](https://md.research.elamur.ai/free/en/authors/anji-liu/), Haowei Lin, [Jiaqi Li](https://md.research.elamur.ai/free/en/authors/jiaqi-li/), [Xiaojian Ma](https://md.research.elamur.ai/free/en/authors/xiaojian-ma/) and [Yitao Liang](https://md.research.elamur.ai/free/en/authors/yitao-liang/) · **Year:** 2024 · **Venue:** arXiv:2403.05313 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2403.05313) · [PDF](https://arxiv.org/pdf/2403.05313) · [DOI 10.48550/arxiv.2403.05313](https://doi.org/10.48550/arxiv.2403.05313) · [arXiv 2403.05313](https://arxiv.org/abs/2403.05313) · [OpenAlex](https://openalex.org/W4392678293)

**Citations:** 10 citations (OpenAlex, 2026-10-14) · [36](https://md.research.elamur.ai/free/en/papers/wang-2024-rat-retrieval-augmented-thoughts-elicit/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Large language models that reason step by step often produce intermediate thoughts containing invented facts. The problem grows in long, multi-step generation such as writing code, planning actions or solving math word problems. The authors propose Retrieval Augmented Thoughts (RAT), a zero-shot prompting scheme that combines chain-of-thought (CoT) prompting with retrieval-augmented generation (RAG). The model first drafts a complete step-by-step plan on its own. It then moves through the steps in order. For each one, it builds a search query from the task, the already-corrected steps and the current step, fetches supporting documents, and rewrites only that step before continuing. No model training is involved. The authors tested RAT with GPT-3.5, GPT-4 and CodeLLaMA-7b on code generation, mathematical reasoning, Minecraft task planning and creative writing. It consistently outperformed plain CoT and standard RAG, and ablations showed that both retrieval-based revision and step-wise progression contribute. The method is useful for building grounded generation pipelines, but its benefit depends on the base model's strength and the quality of the knowledge base.

## Key points
- Compared with baselines, RAT delivered average relative improvements of 13.63% in code generation, 16.96% in math reasoning, 42.78% in embodied task planning and 19.2% in creative writing.
- Code benchmarks: HumanEval +20.94%, HumanEval+ +18.89%, MBPP +14.83%, MBPP+ +1.86%.
- Math benchmarks: GSM8K +8.36% and GSMHard +31.37%. In Minecraft planning, executability rose 2.96 times and plausibility rose by 51.94%. Human ratings of creative writing improved by 19.19%.
- Retrieval ranks documents by cosine similarity of text embeddings (e.g., Sentence-BERT-style models). Queries come from the draft reasoning, not only from the original prompt.
- Stated limitations:
  - RAT relies on the base LLM's CoT and in-context abilities, so its effect on weak models like GPT-2 is uncertain.
  - It needs a relevant, well-maintained knowledge base.
  - Strictly linear step-by-step reasoning may be redundant for simple questions, or too limited for problems that need tree-like reasoning.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 315 works · 43,267 citations
- [University of California, Los Angeles](https://md.research.elamur.ai/free/en/universities/university-of-california-los-angeles/) — University, Los Angeles, United States · 99 works · 56,333 citations
- [Beijing Institute for General Artificial Intelligence](https://md.research.elamur.ai/free/en/government/beijing-institute-for-general-artificial-intelligence/) — Government body, Beijing, China · 11 works · 633 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (2): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works · [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 552 works

## Cite

Cite as: Zihao Wang, Anji Liu, Haowei Lin, Jiaqi Li, Xiaojian Ma and Yitao Liang. 2024. RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Horizon Generation. arXiv:2403.05313.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{wang2024rat,
  title = {{RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Horizon Generation}},
  author = {Zihao Wang, Anji Liu, Haowei Lin, Jiaqi Li, Xiaojian Ma and Yitao Liang},
  year = {2024},
  doi = {10.48550/arxiv.2403.05313},
  eprint = {2403.05313},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2403.05313}
}
```

- [← previous Language models scale reliably with over-training and on downstream tasks](https://md.research.elamur.ai/free/en/papers/gadre-2024-language-models-scale-reliably-over/)
- [next → Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies](https://md.research.elamur.ai/free/en/papers/tao-2024-scaling-laws-vocabulary-larger-models/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/)

## References

68 references, 36 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. A. Asai, Z. Wu, Y. Wang, A. Sil, and H. Hajishirzi. [Self-rag: Learning to retrieve, generate, and critique through self-reflection](https://md.research.elamur.ai/free/en/papers/asai-2023-self-rag-learning-retrieve-generate/). arXiv preprint arXiv:2310.11511, 2023.
2. J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv:2108.07732, 2021.
3. J. Baek, A. F. Aji, and A. Saffari. [Knowledge-augmented language model prompting for zero-shot knowledge graph question answering](https://md.research.elamur.ai/free/en/papers/baek-2023-knowledge-augmented-language-model-prompting/). arXiv preprint arXiv:2306.04136, 2023.
4. B. Baker, I. Akkaya, P. Zhokhov, J. Huizinga, J. Tang, A. Ecoffet, B. Houghton, R. Sampedro, and J. Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. arXiv preprint arXiv:2206.11795, 2022. — [arXiv:2206.11795](https://arxiv.org/abs/2206.11795) [find](https://scholar.google.com/scholar?q=B%2E+Baker%2C+I%2E+Akkaya%2C+P%2E+Zhokhov%2C+J%2E+Huizinga%2C+J%2E+Tang%2C+A%2E+Ecoffet%2C+B%2E+Houghton%2C+R%2E+Sampedro%2C+and+J%2E+Clune%2E+Video+pretraining+%28vpt%29%3A+Learning+to+act+by+watching+unlabeled+online+videos%2E+arXiv+preprint+)
5. M. Besta, N. Blach, A. Kubicek, R. Gerstenberger, L. Gianinazzi, J. Gajda, T. Lehmann, M. Podstawski, H. Niewiadomski, P. Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. arXiv preprint arXiv:2308.09687, 2023. [find](https://scholar.google.com/scholar?q=M%2E+Besta%2C+N%2E+Blach%2C+A%2E+Kubicek%2C+R%2E+Gerstenberger%2C+L%2E+Gianinazzi%2C+J%2E+Gajda%2C+T%2E+Lehmann%2C+M%2E+Podstawski%2C+H%2E+Niewiadomski%2C+P%2E+Nyczyk%2C+et+al%2E+Graph+of+thoughts%3A+Solving+elaborate+problems+with+large+langua)
6. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901, 2020.
7. S. Cai, Z. Wang, X. Ma, A. Liu, and Y. Liang. Open-world multi-task control through goal-aware representation learning and adaptive horizon prediction. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13734–13744, 2023a. [find](https://scholar.google.com/scholar?q=S%2E+Cai%2C+Z%2E+Wang%2C+X%2E+Ma%2C+A%2E+Liu%2C+and+Y%2E+Liang%2E+Open%2Dworld+multi%2Dtask+control+through+goal%2Daware+representation+learning+and+adaptive+horizon+prediction%2E+2023+IEEE%2FCVF+Conference+on+Computer+Vision+and+)
8. S. Cai, B. Zhang, Z. Wang, X. Ma, A. Liu, and Y. Liang. Groot: Learning to follow instructions by watching gameplay videos. arXiv preprint arXiv:2310.08235, 2023b. — [arXiv:2310.08235](https://arxiv.org/abs/2310.08235) [find](https://scholar.google.com/scholar?q=S%2E+Cai%2C+B%2E+Zhang%2C+Z%2E+Wang%2C+X%2E+Ma%2C+A%2E+Liu%2C+and+Y%2E+Liang%2E+Groot%3A+Learning+to+follow+instructions+by+watching+gameplay+videos%2E+arXiv+preprint+arXiv%3A2310%2E08235%2C+2023b%2E+%E2%80%94+%5BarXiv%3A2310%2E08235%5D%28https%3A%2F%2Farxiv%2Eo)
9. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374, 2021.
10. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168, 2021.
11. A. Creswell and M. Shanahan. [Faithful reasoning using large language models](https://md.research.elamur.ai/free/en/papers/creswell-2022-faithful-reasoning-using-large-language/). arXiv preprint arXiv:2208.14271, 2022.
12. A. Creswell, M. Shanahan, and I. Higgins. Selection-inference: Exploiting large language models for interpretable logical reasoning. arXiv preprint arXiv:2205.09712, 2022. [find](https://scholar.google.com/scholar?q=A%2E+Creswell%2C+M%2E+Shanahan%2C+and+I%2E+Higgins%2E+Selection%2Dinference%3A+Exploiting+large+language+models+for+interpretable+logical+reasoning%2E+arXiv+preprint+arXiv%3A2205%2E09712%2C+2022%2E)
13. S. Dhuliawala, M. Komeili, J. Xu, R. Raileanu, X. Li, A. Celikyilmaz, and J. Weston. [Chain-of-verification reduces hallucination in large language models](https://md.research.elamur.ai/free/en/papers/dhuliawala-2023-chain-verification-reduces-hallucination-large/). arXiv preprint arXiv: 2309.11495, 2023.
14. Z. Feng, X. Feng, D. Zhao, M. Yang, and B. Qin. Retrieval-generation synergy augmented large language models. ArXiv, abs/2310.05149, 2023a. — [arXiv:2310.05149](https://arxiv.org/abs/2310.05149) [find](https://scholar.google.com/scholar?q=Z%2E+Feng%2C+X%2E+Feng%2C+D%2E+Zhao%2C+M%2E+Yang%2C+and+B%2E+Qin%2E+Retrieval%2Dgeneration+synergy+augmented+large+language+models%2E+ArXiv%2C+abs%2F2310%2E05149%2C+2023a%2E+%E2%80%94+%5BarXiv%3A2310%2E05149%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2310%2E05149%29)
15. Z. Feng, X. Feng, D. Zhao, M. Yang, and B. Qin. Retrieval-generation synergy augmented large language models. arXiv preprint arXiv:2310.05149, 2023b. — [arXiv:2310.05149](https://arxiv.org/abs/2310.05149) [find](https://scholar.google.com/scholar?q=Z%2E+Feng%2C+X%2E+Feng%2C+D%2E+Zhao%2C+M%2E+Yang%2C+and+B%2E+Qin%2E+Retrieval%2Dgeneration+synergy+augmented+large+language+models%2E+arXiv+preprint+arXiv%3A2310%2E05149%2C+2023b%2E+%E2%80%94+%5BarXiv%3A2310%2E05149%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2310%2E05)
16. L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig. Pal: Program-aided language models. arXiv preprint arXiv:2211.10435, 2022. [find](https://scholar.google.com/scholar?q=L%2E+Gao%2C+A%2E+Madaan%2C+S%2E+Zhou%2C+U%2E+Alon%2C+P%2E+Liu%2C+Y%2E+Yang%2C+J%2E+Callan%2C+and+G%2E+Neubig%2E+Pal%3A+Program%2Daided+language+models%2E+arXiv+preprint+arXiv%3A2211%2E10435%2C+2022%2E)
17. Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, and H. Wang. [Retrieval-augmented generation for large language models: A survey](https://md.research.elamur.ai/free/en/papers/gao-2023-retrieval-augmented-generation-large-language/). arXiv preprint arXiv:2312.10997, 2023.
18. J. Gu, Y. Wang, K. Cho, and V. O. Li. Search engine guided neural machine translation. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018. [find](https://scholar.google.com/scholar?q=J%2E+Gu%2C+Y%2E+Wang%2C+K%2E+Cho%2C+and+V%2E+O%2E+Li%2E+Search+engine+guided+neural+machine+translation%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+2018%2E)
19. D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y. Wu, Y. K. Li, F. Luo, Y. Xiong, and W. Liang. Deepseek-coder: When the large language model meets programming – the rise of code intelligence. arXiv preprint arXiv:2401.14196, 2024. [find](https://scholar.google.com/scholar?q=D%2E+Guo%2C+Q%2E+Zhu%2C+D%2E+Yang%2C+Z%2E+Xie%2C+K%2E+Dong%2C+W%2E+Zhang%2C+G%2E+Chen%2C+X%2E+Bi%2C+Y%2E+Wu%2C+Y%2E+K%2E+Li%2C+F%2E+Luo%2C+Y%2E+Xiong%2C+and+W%2E+Liang%2E+Deepseek%2Dcoder%3A+When+the+large+language+model+meets+programming+%E2%80%93+the+rise+of+code+)
20. R. Herbrich, T. Minka, and T. Graepel. Trueskill™: a bayesian skill rating system. Advances in neural information processing systems, 19, 2006. [find](https://scholar.google.com/scholar?q=R%2E+Herbrich%2C+T%2E+Minka%2C+and+T%2E+Graepel%2E+Trueskill%E2%84%A2%3A+a+bayesian+skill+rating+system%2E+Advances+in+neural+information+processing+systems%2C+19%2C+2006%2E)
21. K. J. Holyoak and R. G. Morrison. The Oxford handbook of thinking and reasoning. Oxford University Press, 2012. [find](https://scholar.google.com/scholar?q=K%2E+J%2E+Holyoak+and+R%2E+G%2E+Morrison%2E+The+Oxford+handbook+of+thinking+and+reasoning%2E+Oxford+University+Press%2C+2012%2E)
22. W. Huang, P. Abbeel, D. Pathak, and I. Mordatch. [Language models as zero-shot planners: Extracting actionable knowledge for embodied agents](https://md.research.elamur.ai/free/en/papers/huang-2022-language-models-zero-shot-planners/). ICML, 2022.
23. Z. Ke, H. Lin, Y. Shao, H. Xu, L. Shu, and B. Liu. Continual training of language models for few-shot learning. arXiv preprint arXiv:2210.05549, 2022a. [find](https://scholar.google.com/scholar?q=Z%2E+Ke%2C+H%2E+Lin%2C+Y%2E+Shao%2C+H%2E+Xu%2C+L%2E+Shu%2C+and+B%2E+Liu%2E+Continual+training+of+language+models+for+few%2Dshot+learning%2E+arXiv+preprint+arXiv%3A2210%2E05549%2C+2022a%2E)
24. Z. Ke, Y. Shao, H. Lin, H. Xu, L. Shu, and B. Liu. Adapting a language model while preserving its general knowledge. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10177–10188, 2022b. [find](https://scholar.google.com/scholar?q=Z%2E+Ke%2C+Y%2E+Shao%2C+H%2E+Lin%2C+H%2E+Xu%2C+L%2E+Shu%2C+and+B%2E+Liu%2E+Adapting+a+language+model+while+preserving+its+general+knowledge%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+Natural+Language+Proc)
25. Z. Ke, Y. Shao, H. Lin, T. Konishi, G. Kim, and B. Liu. Continual pre-training of language models. In The Eleventh International Conference on Learning Representations, 2023. [find](https://scholar.google.com/scholar?q=Z%2E+Ke%2C+Y%2E+Shao%2C+H%2E+Lin%2C+T%2E+Konishi%2C+G%2E+Kim%2C+and+B%2E+Liu%2E+Continual+pre%2Dtraining+of+language+models%2E+In+The+Eleventh+International+Conference+on+Learning+Representations%2C+2023%2E)
26. T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). Advances in neural information processing systems, 35:22199–22213, 2022.
27. P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, et al. [Retrieval-augmented generation for knowledge-intensive nlp tasks](https://md.research.elamur.ai/free/en/papers/lewis-2020-retrieval-augmented-generation-knowledge-intensive/). Advances in Neural Information Processing Systems, 33:9459–9474, 2020a.
28. P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, et al. [Retrieval-augmented generation for knowledge-intensive nlp tasks](https://md.research.elamur.ai/free/en/papers/lewis-2020-retrieval-augmented-generation-knowledge-intensive/). Advances in Neural Information Processing Systems, 33:9459–9474, 2020b.
29. C. Li, J. Liang, A. Zeng, X. Chen, K. Hausman, D. Sadigh, S. Levine, L. Fei-Fei, F. Xia, and B. Ichter. Chain of code: Reasoning with a language model-augmented code emulator, 2023a. [find](https://scholar.google.com/scholar?q=C%2E+Li%2C+J%2E+Liang%2C+A%2E+Zeng%2C+X%2E+Chen%2C+K%2E+Hausman%2C+D%2E+Sadigh%2C+S%2E+Levine%2C+L%2E+Fei%2DFei%2C+F%2E+Xia%2C+and+B%2E+Ichter%2E+Chain+of+code%3A+Reasoning+with+a+language+model%2Daugmented+code+emulator%2C+2023a%2E)
30. X. Li, R. Zhao, Y. K. Chia, B. Ding, S. Joty, S. Poria, and L. Bing. [Chain-of-knowledge: Grounding large language models via dynamic knowledge adapting over heterogeneous sources](https://md.research.elamur.ai/free/en/papers/li-2023-chain-knowledge-grounding-large-language/). In The Twelfth International Conference on Learning Representations, 2023b.
31. S. Lifshitz, K. Paster, H. Chan, J. Ba, and S. McIlraith. Steve-1: A generative model for text-to-behavior in minecraft. arXiv preprint arXiv:2306.00937, 2023. — [arXiv:2306.00937](https://arxiv.org/abs/2306.00937) [find](https://scholar.google.com/scholar?q=S%2E+Lifshitz%2C+K%2E+Paster%2C+H%2E+Chan%2C+J%2E+Ba%2C+and+S%2E+McIlraith%2E+Steve%2D1%3A+A+generative+model+for+text%2Dto%2Dbehavior+in+minecraft%2E+arXiv+preprint+arXiv%3A2306%2E00937%2C+2023%2E+%E2%80%94+%5BarXiv%3A2306%2E00937%5D%28https%3A%2F%2Farxiv%2Eorg%2Fa)
32. H. Lin, Z. Wang, J. Ma, and Y. Liang. Mcu: A task-centric framework for open-ended agent evaluation in minecraft. arXiv preprint arXiv:2310.08367, 2023. — [arXiv:2310.08367](https://arxiv.org/abs/2310.08367) [find](https://scholar.google.com/scholar?q=H%2E+Lin%2C+Z%2E+Wang%2C+J%2E+Ma%2C+and+Y%2E+Liang%2E+Mcu%3A+A+task%2Dcentric+framework+for+open%2Dended+agent+evaluation+in+minecraft%2E+arXiv+preprint+arXiv%3A2310%2E08367%2C+2023%2E+%E2%80%94+%5BarXiv%3A2310%2E08367%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2310)
33. H. Lin, B. Huang, H. Ye, Q. Chen, Z. Wang, S. Li, J. Ma, X. Wan, J. Zou, and Y. Liang. Selecting large language model to fine-tune via rectified scaling law. arXiv preprint arXiv:2402.02314, 2024. — [arXiv:2402.02314](https://arxiv.org/abs/2402.02314) [find](https://scholar.google.com/scholar?q=H%2E+Lin%2C+B%2E+Huang%2C+H%2E+Ye%2C+Q%2E+Chen%2C+Z%2E+Wang%2C+S%2E+Li%2C+J%2E+Ma%2C+X%2E+Wan%2C+J%2E+Zou%2C+and+Y%2E+Liang%2E+Selecting+large+language+model+to+fine%2Dtune+via+rectified+scaling+law%2E+arXiv+preprint+arXiv%3A2402%2E02314%2C+2024%2E+%E2%80%94+%5B)
34. C. Liu, X. Li, L. Shang, X. Jiang, Q. Liu, E. Y. Lam, and N. Wong. Gradually excavating external knowledge for implicit complex question answering. In Conference on Empirical Methods in Natural Language Processing, 2023a. [find](https://scholar.google.com/scholar?q=C%2E+Liu%2C+X%2E+Li%2C+L%2E+Shang%2C+X%2E+Jiang%2C+Q%2E+Liu%2C+E%2E+Y%2E+Lam%2C+and+N%2E+Wong%2E+Gradually+excavating+external+knowledge+for+implicit+complex+question+answering%2E+In+Conference+on+Empirical+Methods+in+Natural+Langua)
35. J. Liu, C. S. Xia, Y. Wang, and L. Zhang. [Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation](https://md.research.elamur.ai/free/en/papers/liu-2023-your-code-generated-chatgpt-really/). In Thirty-seventh Conference on Neural Information Processing Systems, 2023b.
36. S. Lu, N. Duan, H. Han, D. Guo, S.-w. Hwang, and A. Svyatkovskiy. [Reacc: A retrieval-augmented code completion framework](https://md.research.elamur.ai/free/en/papers/lu-2022-reacc-retrieval-augmented-code-completion/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6227–6240, 2022.
37. N. Nashid, M. Sintaha, and A. Mesbah. Retrieval-based prompt selection for code-related few-shot learning. In Proceedings of the 45th International Conference on Software Engineering (ICSE’23), 2023. [find](https://scholar.google.com/scholar?q=N%2E+Nashid%2C+M%2E+Sintaha%2C+and+A%2E+Mesbah%2E+Retrieval%2Dbased+prompt+selection+for+code%2Drelated+few%2Dshot+learning%2E+In+Proceedings+of+the+45th+International+Conference+on+Software+Engineering+%28ICSE%E2%80%9923%29%2C+2023%2E)
38. M. Nye, A. J. Andreassen, G. Gur-Ari, H. Michalewski, J. Austin, D. Bieber, D. Dohan, A. Lewkowycz, M. Bosma, D. Luan, et al. [Show your work: Scratchpads for intermediate computation with language models](https://md.research.elamur.ai/free/en/papers/al-2021-show-your-work-scratchpads-intermediate/). arXiv preprint arXiv:2112.00114, 2021.
39. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/), 2023.
40. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). arXiv preprint arXiv:2203.02155, 2022.
41. V. Rawte, A. Sheth, and A. Das. [A survey of hallucination in large foundation models](https://md.research.elamur.ai/free/en/papers/rawte-2023-survey-hallucination-large-foundation-models/). arXiv preprint arXiv:2309.05922, 2023.
42. N. Reimers and I. Gurevych. [Sentence-bert: Sentence embeddings using siamese bert-networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/). arXiv preprint arXiv:1908.10084, 2019.
43. D. Ribeiro, S. Wang, X. Ma, R. Dong, X. Wei, H. Zhu, X. Chen, Z. Huang, P. Xu, A. Arnold, et al. Entailment tree explanations via iterative retrieval-generation reasoner. arXiv preprint arXiv:2205.09224, 2022. [find](https://scholar.google.com/scholar?q=D%2E+Ribeiro%2C+S%2E+Wang%2C+X%2E+Ma%2C+R%2E+Dong%2C+X%2E+Wei%2C+H%2E+Zhu%2C+X%2E+Chen%2C+Z%2E+Huang%2C+P%2E+Xu%2C+A%2E+Arnold%2C+et+al%2E+Entailment+tree+explanations+via+iterative+retrieval%2Dgeneration+reasoner%2E+arXiv+preprint+arXiv%3A2205%2E092)
44. B. Rozière, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. Tan, Y. Adi, J. Liu, T. Remez, J. Rapin, A. Kozhevnikov, I. Evtimov, J. Bitton, M. P. Bhatt, C. C. Ferrer, A. Grattafiori, W. Xiong, A. D’efossez, J. Copet, F. Azhar, H. Touvron, L. Martin, N. Usunier, T. Scialom, and G. Synnaeve. Code llama… — [Code Llama: Open Foundation Models for Code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/)
45. N. Shinn, B. Labash, and A. Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.[11366, 2023](https://md.research.elamur.ai/free/en/papers/shinn-2023-reflexion-language-agents-verbal-reinforcement/).
46. S. Siriwardhana, R. Weerasekera, E. Wen, T. Kaluarachchi, R. Rana, and S. Nanayakkara. [Improving the domain adaptation of retrieval augmented generation (rag) models for open domain question answering](https://md.research.elamur.ai/free/en/papers/siriwardhana-2023-improving-domain-adaptation-retrieval-augmented/). Transactions of the Association for Computational Linguistics, 11: 1–17, 2023.
47. G. P. Team. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). arXiv preprint arXiv: 2204.02311, 2022.
48. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023.
49. H. Trivedi, N. Balasubramanian, T. Khot, and A. Sabharwal. [Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions](https://md.research.elamur.ai/free/en/papers/trivedi-2022-interleaving-retrieval-chain-thought-reasoning/). ArXiv, abs/2212.10509, 2022a.
50. H. Trivedi, N. Balasubramanian, T. Khot, and A. Sabharwal. [Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions](https://md.research.elamur.ai/free/en/papers/trivedi-2022-interleaving-retrieval-chain-thought-reasoning/). arXiv preprint arXiv:2212.10509, 2022b.
51. X. Wang and D. Zhou. Chain-of-thought reasoning without prompting. arXiv preprint arXiv:2402.10200, 2024. [find](https://scholar.google.com/scholar?q=X%2E+Wang+and+D%2E+Zhou%2E+Chain%2Dof%2Dthought+reasoning+without+prompting%2E+arXiv+preprint+arXiv%3A2402%2E10200%2C+2024%2E)
52. X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). arXiv preprint arXiv:2203.11171, 2022.
53. X. Wang, J. Wei, D. Schuurmans, Q. V. Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). In The Eleventh International Conference on Learning Representations, ICLR 2023, 2023a.
54. Z. Wang, S. Cai, A. Liu, Y. Jin, J. Hou, B. Zhang, H. Lin, Z. He, Z. Zheng, Y. Yang, X. Ma, and Y. Liang. Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models. ArXiv, abs/2311.05997, 2023b. [find](https://scholar.google.com/scholar?q=Z%2E+Wang%2C+S%2E+Cai%2C+A%2E+Liu%2C+Y%2E+Jin%2C+J%2E+Hou%2C+B%2E+Zhang%2C+H%2E+Lin%2C+Z%2E+He%2C+Z%2E+Zheng%2C+Y%2E+Yang%2C+X%2E+Ma%2C+and+Y%2E+Liang%2E+Jarvis%2D1%3A+Open%2Dworld+multi%2Dtask+agents+with+memory%2Daugmented+multimodal+language+models%2E+ArXiv)
55. Z. Wang, S. Cai, A. Liu, X. Ma, and Y. Liang. [Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents](https://md.research.elamur.ai/free/en/papers/wang-2023-describe-explain-plan-select-interactive/). arXiv preprint arXiv:2302.01560, 2023c.
56. J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. 36th Conference on Neural Information Processing Systems (NeurIPS 2022), 2022. — [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
57. Z. Wen, Z. Tian, W. Wu, Y. Yang, Y. Shi, Z. Huang, and D. Li. Grove: a retrieval-augmented complex story generation framework with a forest of evidence. arXiv preprint arXiv:2310.05388, 2023. [find](https://scholar.google.com/scholar?q=Z%2E+Wen%2C+Z%2E+Tian%2C+W%2E+Wu%2C+Y%2E+Yang%2C+Y%2E+Shi%2C+Z%2E+Huang%2C+and+D%2E+Li%2E+Grove%3A+a+retrieval%2Daugmented+complex+story+generation+framework+with+a+forest+of+evidence%2E+arXiv+preprint+arXiv%3A2310%2E05388%2C+2023%2E)
58. S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao. [React: Synergizing reasoning and acting in language models](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/). arXiv preprint arXiv:2210.03629, 2022.
59. D. Zhou, N. Schärli, L. Hou, J. Wei, N. Scales, X. Wang, D. Schuurmans, C. Cui, O. Bousquet, Q. Le, et al. [Least-to-most prompting enables complex reasoning in large language models](https://md.research.elamur.ai/free/en/papers/al-2022-least-most-prompting-enables-complex/). arXiv preprint arXiv:2205.10625, 2022a.
60. D. Zhou, N. Schärli, L. Hou, J. Wei, N. Scales, X. Wang, D. Schuurmans, C. Cui, O. Bousquet, Q. V. Le, and E. H. Chi. [Least-to-most prompting enables complex reasoning in large language models](https://md.research.elamur.ai/free/en/papers/al-2022-least-most-prompting-enables-complex/). In The Eleventh International Conference on Learning Representations, ICLR 2023, 2023.
61. S. Zhou, U. Alon, F. F. Xu, Z. Jiang, and G. Neubig. Docprompting: Generating code by retrieving the docs. In The Eleventh International Conference on Learning Representations, 2022b. [find](https://scholar.google.com/scholar?q=S%2E+Zhou%2C+U%2E+Alon%2C+F%2E+F%2E+Xu%2C+Z%2E+Jiang%2C+and+G%2E+Neubig%2E+Docprompting%3A+Generating+code+by+retrieving+the+docs%2E+In+The+Eleventh+International+Conference+on+Learning+Representations%2C+2022b%2E)
62. S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023. [find](https://scholar.google.com/scholar?q=S%2E+Yao%2C+D%2E+Yu%2C+J%2E+Zhao%2C+I%2E+Shafran%2C+T%2E+L%2E+Griffiths%2C+Y%2E+Cao%2C+and+K%2E+Narasimhan%2E+Tree+of+thoughts%3A+Deliberate+problem+solving+with+large+language+models%2C+2023%2E)
63. H. Yuan, C. Zhang, H. Wang, F. Xie, P. Cai, H. Dong, and Z. Lu. Plan4mc: Skill reinforcement learning and planning for open-world minecraft tasks. arXiv preprint arXiv:2303.16563, 2023. — [arXiv:2303.16563](https://arxiv.org/abs/2303.16563) [find](https://scholar.google.com/scholar?q=H%2E+Yuan%2C+C%2E+Zhang%2C+H%2E+Wang%2C+F%2E+Xie%2C+P%2E+Cai%2C+H%2E+Dong%2C+and+Z%2E+Lu%2E+Plan4mc%3A+Skill+reinforcement+learning+and+planning+for+open%2Dworld+minecraft+tasks%2E+arXiv+preprint+arXiv%3A2303%2E16563%2C+2023%2E+%E2%80%94+%5BarXiv%3A2303%2E)
64. H. Yuan, Z. Mu, F. Xie, and Z. Lu. Pre-training goal-based models for sample-efficient reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024. [find](https://scholar.google.com/scholar?q=H%2E+Yuan%2C+Z%2E+Mu%2C+F%2E+Xie%2C+and+Z%2E+Lu%2E+Pre%2Dtraining+goal%2Dbased+models+for+sample%2Defficient+reinforcement+learning%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2C+2024%2E)
65. E. Zelikman, Y. Wu, J. Mu, and N. Goodman. [Star: Bootstrapping reasoning with reasoning](https://md.research.elamur.ai/free/en/papers/zelikman-2022-star-bootstrapping-reasoning-reasoning/). Advances in Neural Information Processing Systems, 35:15476–15488, 2022.
66. C. Zhang, K. Yang, S. Hu, Z. Wang, G. Li, Y. Sun, C. Zhang, Z. Zhang, A. Liu, S.-C. Zhu, et al. Proagent: Building proactive cooperative ai with large language models. arXiv preprint arXiv:2308.11339, 2023. — [arXiv:2308.11339](https://arxiv.org/abs/2308.11339) [find](https://scholar.google.com/scholar?q=C%2E+Zhang%2C+K%2E+Yang%2C+S%2E+Hu%2C+Z%2E+Wang%2C+G%2E+Li%2C+Y%2E+Sun%2C+C%2E+Zhang%2C+Z%2E+Zhang%2C+A%2E+Liu%2C+S%2E%2DC%2E+Zhu%2C+et+al%2E+Proagent%3A+Building+proactive+cooperative+ai+with+large+language+models%2E+arXiv+preprint+arXiv%3A2308%2E11339%2C)
67. R. Zhao, H. Chen, W. Wang, F. Jiao, X. L. Do, C. Qin, B. Ding, X. Guo, M. Li, X. Li, and S. R. Joty. Retrieving multimodal information for augmented generation: A survey. ArXiv, abs/2303.10868, 2023. [find](https://scholar.google.com/scholar?q=R%2E+Zhao%2C+H%2E+Chen%2C+W%2E+Wang%2C+F%2E+Jiao%2C+X%2E+L%2E+Do%2C+C%2E+Qin%2C+B%2E+Ding%2C+X%2E+Guo%2C+M%2E+Li%2C+X%2E+Li%2C+and+S%2E+R%2E+Joty%2E+Retrieving+multimodal+information+for+augmented+generation%3A+A+survey%2E+ArXiv%2C+abs%2F2303%2E10868%2C+2023%2E)
68. A. Task Details A.1. Code Generation [find](https://scholar.google.com/scholar?q=A%2E+Task+Details+A%2E1%2E+Code+Generation)

*Updated 2 November 2026*