---
title: "Towards a Cleaner Document-Oriented — Abadji 2022"
description: "Earlier OSCAR releases (OSCAR 2019, OSCAR 21.09) built with the Ungoliant pipeline assigned a language to each line of Common Crawl text separately."
authors: ["Julien Abadji", "Pedro Ortiz Suarez", "Laurent Romary", "Benoit Sagot"]
year: 2022
venue: "arXiv:2201.06642"
source: "https://arxiv.org/abs/2201.06642"
doi: "10.48550/arxiv.2201.06642"
arxiv: "2201.06642"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/abadji-2022-towards-cleaner-document-oriented-multilingual/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/abadji-2022-towards-cleaner-document-oriented-multilingual/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/5/) / Towards a Cleaner Document-Oriented Multilingual Crawled Corpus

# Towards a Cleaner Document-Oriented Multilingual Crawled Corpus

**Authors:** Julien Abadji, Pedro Ortiz Suarez, [Laurent Romary](https://md.research.elamur.ai/free/en/authors/laurent-romary/), [Benoit Sagot](https://md.research.elamur.ai/free/en/authors/benoit-sagot/) · **Year:** 2022 · **Venue:** arXiv:2201.06642 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2201.06642) · [PDF](https://arxiv.org/pdf/2201.06642) · [DOI 10.48550/arxiv.2201.06642](https://doi.org/10.48550/arxiv.2201.06642) · [arXiv 2201.06642](https://arxiv.org/abs/2201.06642) · [OpenAlex](https://openalex.org/W4221165895)

**Citations:** 17 citations (openalex-snapshot · 2026-10-14) · [134](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Earlier OSCAR releases (OSCAR 2019, OSCAR 21.09) built with the Ungoliant pipeline assigned a language to each line of Common Crawl text separately. As a result, multilingual web pages were broken into fragments, and the "documents" were really just runs of same-language lines. Abadji et al. rework Ungoliant so that language is assigned per document, as mC4 and CCNet already do. They also add filters that remove boilerplate such as navigation bars and footers while keeping the remaining text intact. Each document's metadata gains automatic annotations that users can use for quality filtering, for example by length.

Line-level FastText predictions are combined into one document label. Pages with a balanced mix of languages are routed to a dedicated multilingual subset. The resulting OSCAR 22.01 is roughly as large as OSCAR 21.09 and has a similar per-language distribution, and it comes with a deduplicated English portion and a line deduplication tool. This matters because generative language models and Digital Humanities work need text that stays coherent and readable as whole documents.

## Key points
- **Two filters.** One removes contiguous short lines only at the start or end of a document. The other discards a document entirely if its short lines outnumber its long ones.
- **Multilingual label.** A document can be tagged multilingual only if it has at least 5 lines and at most 5 languages. Each of its m languages must cover at least 1/(m+1) of its bytes, and unidentified text may cover no more than 1/(m+1).
- **Monolingual label.** Other documents receive their dominant language if the byte-weighted FastText confidence reaches 0.6. Earlier pipelines used 0.8, and the lower value offsets data lost to document-level filtering.
- **Corpus size.** OSCAR 22.01 includes about 3.2 TB of English (≈432M documents) and a ~12 GB multilingual subset. According to Table 2, Ukrainian accounts for 48.8 GB, about 4.56M documents and 2.88B words.
- **Compute cost.** Generating the corpus took 42.6 hours, with reported power consumption of 0.4266 kWh (PUE 1.58).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

134 papers of our library cite it (of the 140673 whose bibliographies we read); 15 of them have a page here, 119 not yet.

- [The Falcon Series of Open Language Models](https://md.research.elamur.ai/free/en/papers/almazrouei-2023-falcon-series-open-language-models/)
- [Harnessing Multilingual Resources to Question Answering in Arabic](https://md.research.elamur.ai/free/en/papers/alnajjar-2022-harnessing-multilingual-resources-question-answering/)
- [Arabic Stable LM: Adapting Stable LM 2 1.6B to Arabic](https://md.research.elamur.ai/free/en/papers/alyafeai-2024-arabic-stable-lm-adapting-stable/)
- [Universal Dependencies for Sindhi](https://md.research.elamur.ai/free/en/papers/bauer-2025-universal-dependencies-sindhi/)
- [A Survey of Corpora for Germanic Low-Resource Languages and Dialects](https://md.research.elamur.ai/free/en/papers/blaschke-2023-survey-corpora-germanic-low-resource/)
- [Analysis of Data Augmentation Methods for Low-Resource Maltese ASR](https://md.research.elamur.ai/free/en/papers/demarco-2021-analysis-data-augmentation-methods-low/)
- [MADLAD-400: A Multilingual And Document-Level Large Audited Dataset](https://md.research.elamur.ai/free/en/papers/kudugunta-2023-madlad-400-multilingual-document-level/)
- [HeRo: RoBERTa and Longformer Hebrew Language Models](https://md.research.elamur.ai/free/en/papers/langstream-2023-shalumov-hero-roberta-longformer-hebrew-language/)
- [Exploring Tokenization Strategies and Vocabulary Sizes for Enhanced Arabic Language Models](https://md.research.elamur.ai/free/en/papers/langstream-2024-alrefaie-exploring-tokenization-strategies-vocabulary-sizes/)
- [HalleluBERT: Let Every Token That Has Meaning Bear Its Weight](https://md.research.elamur.ai/free/en/papers/lingbulk-2026-schmitt-hallelubert-let-every-token-that/)
- [CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages](https://md.research.elamur.ai/free/en/papers/nguyen-2023-culturax-cleaned-enormous-multilingual-dataset/)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://md.research.elamur.ai/free/en/papers/penedo-2024-fineweb-datasets-decanting-web-finest/)
- [ChatGPT MT: Competitive for High- (but Not Low-) Resource Languages](https://md.research.elamur.ai/free/en/papers/robinson-2023-chatgpt-mt-competitive-high-but/)
- [Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning](https://md.research.elamur.ai/free/en/papers/singh-2024-aya-dataset-open-access-collection/)
- [Multilingual Coreference Resolution in Multiparty Dialogue](https://md.research.elamur.ai/free/en/papers/zheng-2023-multilingual-coreference-resolution-multiparty-dialogue/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{abadji2022cleaner,
  title = {{Towards a Cleaner Document-Oriented Multilingual Crawled Corpus}},
  author = {Abadji, Julien and Suarez, Pedro Ortiz and Romary, Laurent and Sagot, Benoit},
  year = {2022},
  doi = {10.48550/arxiv.2201.06642},
  eprint = {2201.06642},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2201.06642}
}
```

- [← previous TimeLMs: Diachronic Language Models from Twitter](https://md.research.elamur.ai/free/en/papers/loureiro-2022-timelms-diachronic-language-models-twitter/)
- [next → Towards a Unified Multi-Dimensional Evaluator for Text Generation](https://md.research.elamur.ai/free/en/papers/zhong-2022-unified-multi-dimensional-evaluator-text/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/5/)

## References

31 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abadji, J., Suárez, P. J. O., Romary, L., and Sagot, B. (2021). Ungoliant: An optimized pipeline for the generation of a very large-scale multilingual web corpus. In Harald Lüngen, et al., editors, Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-9) 2021. Limerick… [find](https://scholar.google.com/scholar?q=Abadji%2C+J%2E%2C+Sua%CC%81rez%2C+P%2E+J%2E+O%2E%2C+Romary%2C+L%2E%2C+and+Sagot%2C+B%2E+%282021%29%2E+Ungoliant%3A+An+optimized+pipeline+for+the+generation+of+a+very+large%2Dscale+multilingual+web+corpus%2E+In+Harald+Lu%CC%88ngen%2C+et+al%2E%2C+editors%2C+)
2. Al-Rfou’, R., Perozzi, B., and Skiena, S. (2013). Polyglot: Distributed word representations for multilingual NLP. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pages 183–192, Sofia, Bulgaria, August. Association for Computational Linguistics. Bañón, M.,… [find](https://scholar.google.com/scholar?q=Al%2DRfou%E2%80%99%2C+R%2E%2C+Perozzi%2C+B%2E%2C+and+Skiena%2C+S%2E+%282013%29%2E+Polyglot%3A+Distributed+word+representations+for+multilingual+NLP%2E+In+Proceedings+of+the+Seventeenth+Conference+on+Computational+Natural+Language+Learni) [our card](https://md.research.elamur.ai/free/en/papers/rfou-2013-polyglot-distributed-word-representations-multilingual/)
3. Barera, M. (2020). Mind the gap: Addressing structural equity and inclusion on wikipedia. [https://rc.library.uta.edu/uta-ir/handle/10106/29572](https://rc.library.uta.edu/uta-ir/handle/10106/29572). [↗](https://rc.library.uta.edu/uta-ir/handle/10106/29572)
4. Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 610–623, New York, NY, USA. Association for Computin… [find](https://scholar.google.com/scholar?q=Bender%2C+E%2E+M%2E%2C+Gebru%2C+T%2E%2C+McMillan%2DMajor%2C+A%2E%2C+and+Shmitchell%2C+S%2E+%282021%29%2E+On+the+dangers+of+stochastic+parrots%3A+Can+language+models+be+too+big%3F+In+Proceedings+of+the+2021+ACM+Conference+on+Fairness%2C+Ac)
5. Black, S., Gao, L., Wang, P., Leahy, C., and Biderman, S. (2021). GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March. If you use this software, please cite it using these metadata. [find](https://scholar.google.com/scholar?q=Black%2C+S%2E%2C+Gao%2C+L%2E%2C+Wang%2C+P%2E%2C+Leahy%2C+C%2E%2C+and+Biderman%2C+S%2E+%282021%29%2E+GPT%2DNeo%3A+Large+Scale+Autoregressive+Language+Modeling+with+Mesh%2DTensorflow%2C+March%2E+If+you+use+this+software%2C+please+cite+it+using+thes)
6. Bojanowski, P., Grave, E., Joulin, A., and Mikolov, T. (2017). Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. [find](https://scholar.google.com/scholar?q=Bojanowski%2C+P%2E%2C+Grave%2C+E%2E%2C+Joulin%2C+A%2E%2C+and+Mikolov%2C+T%2E+%282017%29%2E+Enriching+word+vectors+with+subword+information%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+5%3A135%E2%80%93146%2E)
7. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess… [find](https://scholar.google.com/scholar?q=Brown%2C+T%2E%2C+Mann%2C+B%2E%2C+Ryder%2C+N%2E%2C+Subbiah%2C+M%2E%2C+Kaplan%2C+J%2E+D%2E%2C+Dhariwal%2C+P%2E%2C+Neelakantan%2C+A%2E%2C+Shyam%2C+P%2E%2C+Sastry%2C+G%2E%2C+Askell%2C+A%2E%2C+Agarwal%2C+S%2E%2C+Herbert%2DVoss%2C+A%2E%2C+Krueger%2C+G%2E%2C+Henighan%2C+T%2E%2C+Child%2C+R%2E%2C+Rames)
8. Curran Associates, Inc. Caswell, I., Breiner, T., van Esch, D., and Bapna, A. (2020). Language ID in the wild: Unexpected challenges on the path to a thousand-language web text corpus. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6588–6608, Barcelona, Spain… [find](https://scholar.google.com/scholar?q=Curran+Associates%2C+Inc%2E+Caswell%2C+I%2E%2C+Breiner%2C+T%2E%2C+van+Esch%2C+D%2E%2C+and+Bapna%2C+A%2E+%282020%29%2E+Language+ID+in+the+wild%3A+Unexpected+challenges+on+the+path+to+a+thousand%2Dlanguage+web+text+corpus%2E+In+Proceedings+) [our card](https://md.research.elamur.ai/free/en/papers/kreutzer-2021-quality-glance-audit-web-crawled/)
9. Chelba, C., Mikolov, T., Schuster, M., Ge, Q., Brants, T., Koehn, P., and Robinson, T. (2014). One billion word benchmark for measuring progress in statistical language modeling. In Haizhou Li, et al., editors, INTERSPEECH 2014, 15th Annual Conference of the International Speech Communication Associ… [find](https://scholar.google.com/scholar?q=Chelba%2C+C%2E%2C+Mikolov%2C+T%2E%2C+Schuster%2C+M%2E%2C+Ge%2C+Q%2E%2C+Brants%2C+T%2E%2C+Koehn%2C+P%2E%2C+and+Robinson%2C+T%2E+%282014%29%2E+One+billion+word+benchmark+for+measuring+progress+in+statistical+language+modeling%2E+In+Haizhou+Li%2C+et+al%2E) [our card](https://md.research.elamur.ai/free/en/papers/chelba-2013-one-billion-word-benchmark-measuring/)
10. ISCA. Desrochers, S., Paradis, C., and Weaver, V. M. (2016). A validation of dram rapl power measurements. In Proceedings of the Second International Symposium on Memory Systems, MEMSYS ’16, page 455–470, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=ISCA%2E+Desrochers%2C+S%2E%2C+Paradis%2C+C%2E%2C+and+Weaver%2C+V%2E+M%2E+%282016%29%2E+A+validation+of+dram+rapl+power+measurements%2E+In+Proceedings+of+the+Second+International+Symposium+on+Memory+Systems%2C+MEMSYS+%E2%80%9916%2C+page+455%E2%80%93)
11. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (… [find](https://scholar.google.com/scholar?q=Devlin%2C+J%2E%2C+Chang%2C+M%2E%2DW%2E%2C+Lee%2C+K%2E%2C+and+Toutanova%2C+K%2E+%282019%29%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+North+Americ)
12. Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C. (2020). The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv e-prints, page arXiv:2101.00027, December. [find](https://scholar.google.com/scholar?q=Gao%2C+L%2E%2C+Biderman%2C+S%2E%2C+Black%2C+S%2E%2C+Golding%2C+L%2E%2C+Hoppe%2C+T%2E%2C+Foster%2C+C%2E%2C+Phang%2C+J%2E%2C+He%2C+H%2E%2C+Thite%2C+A%2E%2C+Nabeshima%2C+N%2E%2C+Presser%2C+S%2E%2C+and+Leahy%2C+C%2E+%282020%29%2E+The+Pile%3A+An+800GB+Dataset+of+Diverse+Text+for+Lan)
13. Grave, E., Bojanowski, P., Gupta, P., Joulin, A., and Mikolov, T. (2018). Learning word vectors for 157 languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Grave%2C+E%2E%2C+Bojanowski%2C+P%2E%2C+Gupta%2C+P%2E%2C+Joulin%2C+A%2E%2C+and+Mikolov%2C+T%2E+%282018%29%2E+Learning+word+vectors+for+157+languages%2E+In+Proceedings+of+the+Eleventh+International+Conference+on+Language+Resources+and+Eva) [our card](https://md.research.elamur.ai/free/en/papers/grave-2018-learning-word-vectors-157-languages/)
14. Heafield, K. (2011). KenLM: Faster and smaller language model queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 187–197, Edinburgh, Scotland, July. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Heafield%2C+K%2E+%282011%29%2E+KenLM%3A+Faster+and+smaller+language+model+queries%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translation%2C+pages+187%E2%80%93197%2C+Edinburgh%2C+Scotland%2C+July%2E+Association+fo) [our card](https://md.research.elamur.ai/free/en/papers/heafield-2011-kenlm-faster-smaller-language-model/)
15. Howard, J. and Ruder, S. (2018). Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339, Melbourne, Australia, July. Association for Computational Linguistics. Jou… [find](https://scholar.google.com/scholar?q=Howard%2C+J%2E+and+Ruder%2C+S%2E+%282018%29%2E+Universal+language+model+fine%2Dtuning+for+text+classification%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Lon) [our card](https://md.research.elamur.ai/free/en/papers/joulin-2016-fasttext-zip-compressing-text-classification/)
16. Joulin, A., Grave, E., Bojanowski, P., and Mikolov, T. (2017). Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427– 431, Valencia, Spain, April. Association… [find](https://scholar.google.com/scholar?q=Joulin%2C+A%2E%2C+Grave%2C+E%2E%2C+Bojanowski%2C+P%2E%2C+and+Mikolov%2C+T%2E+%282017%29%2E+Bag+of+tricks+for+efficient+text+classification%2E+In+Proceedings+of+the+15th+Conference+of+the+European+Chapter+of+the+Association+for+Com)
17. Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., and Carlini, N. (2021). Deduplicating Training Data Makes Language Models Better. arXiv e-prints, page arXiv:2107.06499, July. [find](https://scholar.google.com/scholar?q=Lee%2C+K%2E%2C+Ippolito%2C+D%2E%2C+Nystrom%2C+A%2E%2C+Zhang%2C+C%2E%2C+Eck%2C+D%2E%2C+Callison%2DBurch%2C+C%2E%2C+and+Carlini%2C+N%2E+%282021%29%2E+Deduplicating+Training+Data+Makes+Language+Models+Better%2E+arXiv+e%2Dprints%2C+page+arXiv%3A2107%2E06499%2C+Jul)
18. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv e-prints, page arXiv:1907.11692, July. [find](https://scholar.google.com/scholar?q=Liu%2C+Y%2E%2C+Ott%2C+M%2E%2C+Goyal%2C+N%2E%2C+Du%2C+J%2E%2C+Joshi%2C+M%2E%2C+Chen%2C+D%2E%2C+Levy%2C+O%2E%2C+Lewis%2C+M%2E%2C+Zettlemoyer%2C+L%2E%2C+and+Stoyanov%2C+V%2E+%282019%29%2E+RoBERTa%3A+A+Robustly+Optimized+BERT+Pretraining+Approach%2E+arXiv+e%2Dprints%2C+page+a)
19. Luccioni, A. and Viviano, J. (2021). What’s in the box? an analysis of undesirable content in the Common Crawl corpus. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Shor… [find](https://scholar.google.com/scholar?q=Luccioni%2C+A%2E+and+Viviano%2C+J%2E+%282021%29%2E+What%E2%80%99s+in+the+box%3F+an+analysis+of+undesirable+content+in+the+Common+Crawl+corpus%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Li)
20. Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J. (2013). Distributed representations of words and phrases and their compositionality. In C. J. C. Burges, et al., editors, Advances in Neural Information Processing Systems, volume 26. [find](https://scholar.google.com/scholar?q=Mikolov%2C+T%2E%2C+Sutskever%2C+I%2E%2C+Chen%2C+K%2E%2C+Corrado%2C+G%2E+S%2E%2C+and+Dean%2C+J%2E+%282013%29%2E+Distributed+representations+of+words+and+phrases+and+their+compositionality%2E+In+C%2E+J%2E+C%2E+Burges%2C+et+al%2E%2C+editors%2C+Advances+in)
21. Curran Associates, Inc. Mikolov, T., Grave, E., Bojanowski, P., Puhrsch, C., and Joulin, A. (2018). Advances in pre-training distributed word representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May. European Lang… [find](https://scholar.google.com/scholar?q=Curran+Associates%2C+Inc%2E+Mikolov%2C+T%2E%2C+Grave%2C+E%2E%2C+Bojanowski%2C+P%2E%2C+Puhrsch%2C+C%2E%2C+and+Joulin%2C+A%2E+%282018%29%2E+Advances+in+pre%2Dtraining+distributed+word+representations%2E+In+Proceedings+of+the+Eleventh+Internatio) [our card](https://md.research.elamur.ai/free/en/papers/mikolov-2017-advances-pre-training-distributed-word/)
22. Pennington, J., Socher, R., and Manning, C. (2014). GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar, October. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Pennington%2C+J%2E%2C+Socher%2C+R%2E%2C+and+Manning%2C+C%2E+%282014%29%2E+GloVe%3A+Global+vectors+for+word+representation%2E+In+Proceedings+of+the+2014+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2C+pa)
23. Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018). Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (L… [find](https://scholar.google.com/scholar?q=Peters%2C+M%2E+E%2E%2C+Neumann%2C+M%2E%2C+Iyyer%2C+M%2E%2C+Gardner%2C+M%2E%2C+Clark%2C+C%2E%2C+Lee%2C+K%2E%2C+and+Zettlemoyer%2C+L%2E+%282018%29%2E+Deep+contextualized+word+representations%2E+In+Proceedings+of+the+2018+Conference+of+the+North+America)
24. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI Blog, 1:8. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%2C+Wu%2C+J%2E%2C+Child%2C+R%2E%2C+Luan%2C+D%2E%2C+Amodei%2C+D%2E%2C+and+Sutskever%2C+I%2E+%282019%29%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+Blog%2C+1%3A8%2E)
25. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. [find](https://scholar.google.com/scholar?q=Raffel%2C+C%2E%2C+Shazeer%2C+N%2E%2C+Roberts%2C+A%2E%2C+Lee%2C+K%2E%2C+Narang%2C+S%2E%2C+Matena%2C+M%2E%2C+Zhou%2C+Y%2E%2C+Li%2C+W%2E%2C+and+Liu%2C+P%2E+J%2E+%282020%29%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtext+transformer%2E+Journ)
26. Schwartz, R., Dodge, J., Smith, N. A., and Etzioni, O. (2020). Green ai. Commun. ACM, 63(12):54–63, nov. [find](https://scholar.google.com/scholar?q=Schwartz%2C+R%2E%2C+Dodge%2C+J%2E%2C+Smith%2C+N%2E+A%2E%2C+and+Etzioni%2C+O%2E+%282020%29%2E+Green+ai%2E+Commun%2E+ACM%2C+63%2812%29%3A54%E2%80%9363%2C+nov%2E) [our card](https://md.research.elamur.ai/free/en/papers/schwartz-2019-green-ai/)
27. Strubell, E., Ganesh, A., and McCallum, A. (2019). Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645–3650, Florence, Italy, July. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Strubell%2C+E%2E%2C+Ganesh%2C+A%2E%2C+and+McCallum%2C+A%2E+%282019%29%2E+Energy+and+policy+considerations+for+deep+learning+in+NLP%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics) [our card](https://md.research.elamur.ai/free/en/papers/strubell-2019-energy-policy-considerations-deep-learning/)
28. Trinh, T. H. and Le, Q. V. (2018). A Simple Method for Commonsense Reasoning. arXiv e-prints, page arXiv:1806.02847, June. Wenzek, G., Lachaux, M.-A., Conneau, A., Chaudhary, V., Guzmán, F., Joulin, A., and Grave, E. (2020). CCNet: Extracting high quality monolingual datasets from web crawl data. I… [find](https://scholar.google.com/scholar?q=Trinh%2C+T%2E+H%2E+and+Le%2C+Q%2E+V%2E+%282018%29%2E+A+Simple+Method+for+Commonsense+Reasoning%2E+arXiv+e%2Dprints%2C+page+arXiv%3A1806%2E02847%2C+June%2E+Wenzek%2C+G%2E%2C+Lachaux%2C+M%2E%2DA%2E%2C+Conneau%2C+A%2E%2C+Chaudhary%2C+V%2E%2C+Guzma%CC%81n%2C+F%2E%2C+Joulin%2C+)
29. European Language Resources Association. Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., and Raffel, C. (2021). mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Associati… [find](https://scholar.google.com/scholar?q=European+Language+Resources+Association%2E+Xue%2C+L%2E%2C+Constant%2C+N%2E%2C+Roberts%2C+A%2E%2C+Kale%2C+M%2E%2C+Al%2DRfou%2C+R%2E%2C+Siddhant%2C+A%2E%2C+Barua%2C+A%2E%2C+and+Raffel%2C+C%2E+%282021%29%2E+mT5%3A+A+massively+multilingual+pre%2Dtrained+text%2Dto%2Dte)
30. Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R. R., and Le, Q. V. (2019). Xlnet: Generalized autoregressive pretraining for language understanding. In H. Wallach, et al., editors, Advances in Neural Information Processing Systems, volume 32. [find](https://scholar.google.com/scholar?q=Yang%2C+Z%2E%2C+Dai%2C+Z%2E%2C+Yang%2C+Y%2E%2C+Carbonell%2C+J%2E%2C+Salakhutdinov%2C+R%2E+R%2E%2C+and+Le%2C+Q%2E+V%2E+%282019%29%2E+Xlnet%3A+Generalized+autoregressive+pretraining+for+language+understanding%2E+In+H%2E+Wallach%2C+et+al%2E%2C+editors%2C+Advanc) [our card](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/)
31. Curran Associates, Inc. Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., and Fidler, S. (2015). Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 1… [find](https://scholar.google.com/scholar?q=Curran+Associates%2C+Inc%2E+Zhu%2C+Y%2E%2C+Kiros%2C+R%2E%2C+Zemel%2C+R%2E%2C+Salakhutdinov%2C+R%2E%2C+Urtasun%2C+R%2E%2C+Torralba%2C+A%2E%2C+and+Fidler%2C+S%2E+%282015%29%2E+Aligning+books+and+movies%3A+Towards+story%2Dlike+visual+explanations+by+watchin) [our card](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/)