---
title: "TACO: Topics in Algorithmic COde generation dataset (2023)"
description: "TACO therefore offers both a demanding benchmark and labels that support targeted diagnosis and… 13 references, cited by 67 papers here."
authors: ["Rongao Li", "Jie Fu", "Bo-Wen Zhang", "Tao Huang", "Zhihong Sun", "Chen Lyu", "Guang Liu", "Zhi Jin", "Ge Li"]
year: 2023
venue: "arXiv:2312.14852"
source: "https://arxiv.org/abs/2312.14852"
doi: "10.48550/arxiv.2312.14852"
arxiv: "2312.14852"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/li-2023-taco-topics-algorithmic-code-generation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/li-2023-taco-topics-algorithmic-code-generation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/9/) / TACO: Topics in Algorithmic COde generation dataset

# TACO: Topics in Algorithmic COde generation dataset

**Authors:** Rongao Li, [Jie Fu](https://md.research.elamur.ai/free/en/authors/jie-fu/), [Bo-Wen Zhang](https://md.research.elamur.ai/free/en/authors/bo-wen-zhang/), Tao Huang, Zhihong Sun, Chen Lyu, [Guang Liu](https://md.research.elamur.ai/free/en/authors/guang-liu/), [Zhi Jin](https://md.research.elamur.ai/free/en/authors/zhi-jin/) and [Ge Li](https://md.research.elamur.ai/free/en/authors/ge-li/) · **Year:** 2023 · **Venue:** arXiv:2312.14852 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2312.14852) · [PDF](https://arxiv.org/pdf/2312.14852) · [DOI 10.48550/arxiv.2312.14852](https://doi.org/10.48550/arxiv.2312.14852) · [arXiv 2312.14852](https://arxiv.org/abs/2312.14852) · [OpenAlex](https://openalex.org/W4390215418)

**Citations:** 6 citations (OpenAlex, 2026-10-14) · [67](https://md.research.elamur.ai/free/en/papers/li-2023-taco-topics-algorithmic-code-generation/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Benchmarks such as HumanEval and MBPP are close to saturation. They mostly test short, basic functions. Competition-style sets like APPS and CodeContests suffer from duplicates, weak test suites that let wrong code pass, and little detail on which algorithmic abilities a problem demands. TACO, from BAAI and collaborators, is a larger and harder resource for training and evaluating code generation. Problems were scraped from CodeChef, CodeForces, HackerRank and GeeksforGeeks and merged with APPS, CodeContests and Description2code. The authors then stripped comments, removed duplicate code, added extra test cases and checked solutions with unit tests. Each task carries metadata: difficulty, algorithm topics, skill types, time and memory limits, and expected complexity. In evaluation, even GPT-4 solves few of the hardest tasks, and open models do far worse. Fine-tuning on skill-specific subsets gives small gains on some categories. TACO therefore offers both a demanding benchmark and labels that support targeted diagnosis and training of code models.

## Key points
- **Size:** 26,443 problems (25,443 train, 1,000 test) with 1,539,152 verified Python 3 solutions, down from about 2.05M before deduplication.
  - Each problem has about 58 solutions on average.
  - Test cases average 51.6 per training problem and 202.3 per test problem.
- **Labels:** 968 raw tags were merged into 36 algorithm topics, such as Complete Search and Dynamic Programming, plus separate skill-type labels.
- **Difficulty results:** pass@1 falls sharply as problems get harder.
  - GPT-4 scores 31.5 on easy problems and 2.0 on very hard ones.
  - starcoder-15.5b scores 11.6 on easy and 0.18 on very hard.
- **Skill-specific fine-tuning:** skill-specific LoRAs for starcoder-1b beat a single LoRA trained on the whole set for Data Structures, Greedy Algorithms and Sorting.
- **Availability and caveat:** the data is released as BAAI/TACO on Hugging Face, with evaluation scripts in FlagOpen/TACO on GitHub. Because TACO overlaps with APPS and CodeContests, models trained on it should be evaluated on the TACO test set instead.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 315 works · 43,267 citations
- [Beijing Academy of Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/beijing-academy-of-artificial-intelligence/) — Research institute, Beijing, China · 46 works · 8,682 citations
- [Shandong Normal University](https://md.research.elamur.ai/free/en/universities/shandong-normal-university/) — University, Jinan, China · 2 works · 157 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works · [🖥️ Computer Science Applications](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-science-applications/) · 131 works

🧭 Topics (1): [💻 Code & programming languages](https://md.research.elamur.ai/free/en/topics/code-and-programming/) · 168 works

## Cite

Cite as: Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin and Ge Li. 2023. TACO: Topics in Algorithmic COde generation dataset. arXiv:2312.14852.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{li2023taco,
  title = {{TACO: Topics in Algorithmic COde generation dataset}},
  author = {Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin and Ge Li},
  year = {2023},
  doi = {10.48550/arxiv.2312.14852},
  eprint = {2312.14852},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2312.14852}
}
```

- [← previous ScandEval: A Benchmark for Scandinavian Natural Language Processing](https://md.research.elamur.ai/free/en/papers/nielsen-2023-scandeval-benchmark-scandinavian-natural-language/)
- [next → Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance](https://md.research.elamur.ai/free/en/papers/fu-2023-chain-thought-hub-continuous-effort/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/9/)

## References

13 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv:2108.07732, 2021.
2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901, 2020.
3. Rich Caruana. Multitask learning. Machine learning, 28:41–75, 1997. [find](https://scholar.google.com/scholar?q=Rich+Caruana%2E+Multitask+learning%2E+Machine+learning%2C+28%3A41%E2%80%9375%2C+1997%2E)
4. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374, 2021.
5. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. [Measuring coding challenge competence with apps](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-coding-challenge-competence-apps/). arXiv preprint arXiv:2105.09938, 2021.
6. Antti Laaksonen. Competitive programmer’s handbook. Preprint, 5, 2017. [find](https://scholar.google.com/scholar?q=Antti+Laaksonen%2E+Competitive+programmer%E2%80%99s+handbook%2E+Preprint%2C+5%2C+2017%2E)
7. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. [Competition-level code generation with alphacode](https://md.research.elamur.ai/free/en/papers/li-2022-competition-level-code-generation-alphacode/). Science, 378(6624):1092–1097, 2022.
8. Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161, 2023. [find](https://scholar.google.com/scholar?q=Raymond+Li%2C+Loubna+Ben+Allal%2C+Yangtian+Zi%2C+Niklas+Muennighoff%2C+Denis+Kocetkov%2C+Chenghao+Mou%2C+Marc+Marone%2C+Christopher+Akiki%2C+Jia+Li%2C+Jenny+Chim%2C+et+al%2E+Starcoder%3A+may+the+source+be+with+you%21+arXiv+pre)
9. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. [Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation](https://md.research.elamur.ai/free/en/papers/liu-2023-your-code-generated-chatgpt-really/). arXiv preprint arXiv:2305.01210, 2023.
10. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474, 2022. [find](https://scholar.google.com/scholar?q=Erik+Nijkamp%2C+Bo+Pang%2C+Hiroaki+Hayashi%2C+Lifu+Tu%2C+Huan+Wang%2C+Yingbo+Zhou%2C+Silvio+Savarese%2C+and+Caiming+Xiong%2E+Codegen%3A+An+open+large+language+model+for+code+with+multi%2Dturn+program+synthesis%2E+arXiv+pre)
11. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). ArXiv, abs/2303.08774, 2023.
12. Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, I. Evtimov, Joanna Bitton, Manish P Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre D’efossez, Jade Copet, Fai… — [Code Llama: Open Foundation Models for Code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/)
13. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30, 2017.

*Updated 2 November 2026*