---
title: "OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents — Laurençon 2023"
description: "High-performing vision-language systems such as Flamingo and KOSMOS-1 learn from web pages in which pictures and… 89 references, cited by 113 papers here."
authors: ["Hugo Lauren\\ccon", "Lucile Saulnier", "Leo Tronchon", "Stas Bekman", "Amanpreet Singh", "Anton Lozhkov", "Thomas Wang", "Siddharth Karamcheti", "Alexander M. Rush", "Douwe Kiela", "Matthieu Cord", "Victor Sanh"]
year: 2023
venue: "arXiv:2306.16527"
source: "https://arxiv.org/abs/2306.16527"
doi: "10.48550/arxiv.2306.16527"
arxiv: "2306.16527"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lauren-ccon-2023-obelics-open-web-scale-filtered/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lauren-ccon-2023-obelics-open-web-scale-filtered/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/4/) / OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents

# OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents

**Authors:** [Hugo Lauren\\ccon](https://md.research.elamur.ai/free/en/authors/hugo-lauren-ccon/), [Lucile Saulnier](https://md.research.elamur.ai/free/en/authors/lucile-saulnier/), [Leo Tronchon](https://md.research.elamur.ai/free/en/authors/leo-tronchon/), Stas Bekman, [Amanpreet Singh](https://md.research.elamur.ai/free/en/authors/amanpreet-singh/), [Anton Lozhkov](https://md.research.elamur.ai/free/en/authors/anton-lozhkov/), [Thomas Wang](https://md.research.elamur.ai/free/en/authors/thomas-wang/), Siddharth Karamcheti, [Alexander M. Rush](https://md.research.elamur.ai/free/en/authors/alexander-m-rush/), [Douwe Kiela](https://md.research.elamur.ai/free/en/authors/douwe-kiela/), [Matthieu Cord](https://md.research.elamur.ai/free/en/authors/matthieu-cord/) and [Victor Sanh](https://md.research.elamur.ai/free/en/authors/victor-sanh/) · **Year:** 2023 · **Venue:** arXiv:2306.16527 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2306.16527) · [PDF](https://arxiv.org/pdf/2306.16527) · [DOI 10.48550/arxiv.2306.16527](https://doi.org/10.48550/arxiv.2306.16527) · [arXiv 2306.16527](https://arxiv.org/abs/2306.16527) · [OpenAlex](https://openalex.org/W7133216992)

**Citations:** 70 citations (OpenAlex, 2026-10-14) · [113](https://md.research.elamur.ai/free/en/papers/lauren-ccon-2023-obelics-open-web-scale-filtered/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
High-performing vision-language systems such as Flamingo and KOSMOS-1 learn from web pages in which pictures and prose alternate. However, the corpora behind them were never published, and how they were built is poorly documented. This paper addresses that gap with OBELICS, a public English collection drawn from Common Crawl. The authors start from tens of billions of crawled pages and keep only English, deduplicated, high-quality text. They then simplify the raw HTML DOM trees to strip boilerplate, ads and logos while preserving page layout. Images are filtered individually, text is filtered paragraph by paragraph, and whole documents are screened. A final pass drops opted-out and NSFW images and deduplicates images, documents and paragraphs. To validate the resource, they train IDEFICS, Flamingo-style models with 9B and 80B parameters. The larger model is roughly comparable to Flamingo on several benchmarks, and the smaller one beats OpenFlamingo-9B, which used mmc4. Releasing the data, code and models makes this research direction reproducible.

## Key points
- OBELICS contains 141 million English web documents, 353 million images and 115 billion text tokens, taken from 25 Common Crawl dumps (February 2020 to February 2023).
- The pipeline narrows the data in stages:
  - 41.2B raw documents are reduced to 1.1B after language, deduplication and quality filtering.
  - Filtering then leaves 365M documents with 1.4B images.
  - Responsible filtering and deduplication produce the final 141M documents.
- Compared with mmc4 (103M documents, 585M images, 2019 crawl), OBELICS uses newer pages, DOM-based filtering, and extra deduplication at the image, document and paragraph levels.
- In ablations, training on interleaved documents reached the same performance as image-text pairs with about ten times fewer images. Pairs kept a slight edge on captioning, classification and OCR.
- IDEFICS (80B) matched or beat Flamingo on 4 of 8 tasks at 32 shots. IDEFICS-9B scored 56.5 in aggregate versus 55.8 for OpenFlamingo-9B.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

Topics: [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/)

## Cite

Cite as: Hugo Lauren\\ccon, Lucile Saulnier, Leo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord and Victor Sanh. 2023. OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents. arXiv:2306.16527.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{laurenccon2023obelics,
  title = {{OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents}},
  author = {Hugo Laurenccon, Lucile Saulnier, Leo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord and Victor Sanh},
  year = {2023},
  doi = {10.48550/arxiv.2306.16527},
  eprint = {2306.16527},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.16527}
}
```

- [← previous Multilingual Jailbreak Challenges in Large Language Models](https://md.research.elamur.ai/free/en/papers/deng-2023-multilingual-jailbreak-challenges-large-language/)
- [next → Preference Ranking Optimization for Human Alignment](https://md.research.elamur.ai/free/en/papers/song-2023-preference-ranking-optimization-human-alignment/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/4/)

## References

89 references, 35 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abbas, A., K. Tirumala, D. Simig, S. Ganguli, and A. S. Morcos (2023). Semdedup: Data-efficient learning at web-scale through semantic deduplication. [our card](https://md.research.elamur.ai/free/en/papers/abbas-2023-semdedup-data-efficient-learning-web/)
2. Aghajanyan, A., B. Huang, C. Ross, V. Karpukhin, H. Xu, N. Goyal, D. Okhonko, M. Joshi, G. Ghosh, M. Lewis, and L. Zettlemoyer (2022). Cm3: A causal masked multimodal model of the internet. ArXiv abs/2201.07520. — [arXiv:2201.07520](https://arxiv.org/abs/2201.07520) [find](https://scholar.google.com/scholar?q=Aghajanyan%2C+A%2E%2C+B%2E+Huang%2C+C%2E+Ross%2C+V%2E+Karpukhin%2C+H%2E+Xu%2C+N%2E+Goyal%2C+D%2E+Okhonko%2C+M%2E+Joshi%2C+G%2E+Ghosh%2C+M%2E+Lewis%2C+and+L%2E+Zettlemoyer+%282022%29%2E+Cm3%3A+A+causal+masked+multimodal+model+of+the+internet%2E+ArXiv+abs%2F)
3. Alayrac, J.-B., J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han, Z. Gong, S. Samangooei, M. Monteiro, J. L. Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. a. Bińkowski, R. Barreira, O. Vinyals,… [our card](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/)
4. Antol, S., A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh (2015). VQA: Visual Question Answering. In International Conference on Computer Vision (ICCV). [find](https://scholar.google.com/scholar?q=Antol%2C+S%2E%2C+A%2E+Agrawal%2C+J%2E+Lu%2C+M%2E+Mitchell%2C+D%2E+Batra%2C+C%2E+L%2E+Zitnick%2C+and+D%2E+Parikh+%282015%29%2E+VQA%3A+Visual+Question+Answering%2E+In+International+Conference+on+Computer+Vision+%28ICCV%29%2E)
5. Awadalla, A., I. Gao, J. Gardner, J. Hessel, Y. Hanafy, W. Zhu, K. Marathe, Y. Bitton, S. Gadre, S. Sagawa, J. Jitsev, S. Kornblith, P. W. Koh, G. Ilharco, M. Wortsman, and L. Schmidt (2023). Openflamingo: An open-source framework for training large autoregressive vision-language models. [our card](https://md.research.elamur.ai/free/en/papers/awadalla-2023-openflamingo-open-source-framework-training/)
6. Bai, Y., A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, N. Joseph, S. Kadavath, J. Kernion, T. Conerly, S. El-Showk, N. Elhage, Z. Hatfield-Dodds, D. Hernandez, T. Hume, S. Johnston, S. Kravec, L. Lovitt, N. Nanda, C. Olsson, D. Amodei, T. Brown, J… [find](https://scholar.google.com/scholar?q=Bai%2C+Y%2E%2C+A%2E+Jones%2C+K%2E+Ndousse%2C+A%2E+Askell%2C+A%2E+Chen%2C+N%2E+DasSarma%2C+D%2E+Drain%2C+S%2E+Fort%2C+D%2E+Ganguli%2C+T%2E+Henighan%2C+N%2E+Joseph%2C+S%2E+Kadavath%2C+J%2E+Kernion%2C+T%2E+Conerly%2C+S%2E+El%2DShowk%2C+N%2E+Elhage%2C+Z%2E+Hatfield%2DDodds%2C+D)
7. Beaumont, R. (2021). img2dataset: Easily turn large sets of image urls to an image dataset. [https://github.com/rom1504/img2dataset](https://github.com/rom1504/img2dataset). [↗](https://github.com/rom1504/img2dataset)
8. Bender, E. M., T. Gebru, A. McMillan-Major, and S. Shmitchell (2021). On the dangers of stochastic parrots: Can language models be too big? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. [find](https://scholar.google.com/scholar?q=Bender%2C+E%2E+M%2E%2C+T%2E+Gebru%2C+A%2E+McMillan%2DMajor%2C+and+S%2E+Shmitchell+%282021%29%2E+On+the+dangers+of+stochastic+parrots%3A+Can+language+models+be+too+big%3F+Proceedings+of+the+2021+ACM+Conference+on+Fairness%2C+Accounta)
9. Biderman, S. and W. J. Scheirer (2020, 12 Dec). Pitfalls in machine learning research: Reexamining the development cycle. In J. Zosa Forde, F. Ruiz, M. F. Pradier, and A. Schein (Eds.), Proceedings on "I Can’t Believe It’s Not Better!" at NeurIPS Workshops, Volume 137 of Proceedings of Machine Learn… [find](https://scholar.google.com/scholar?q=Biderman%2C+S%2E+and+W%2E+J%2E+Scheirer+%282020%2C+12+Dec%29%2E+Pitfalls+in+machine+learning+research%3A+Reexamining+the+development+cycle%2E+In+J%2E+Zosa+Forde%2C+F%2E+Ruiz%2C+M%2E+F%2E+Pradier%2C+and+A%2E+Schein+%28Eds%2E%29%2C+Proceedings+on)
10. Biderman, S., H. Schoelkopf, Q. Anthony, H. Bradley, K. O’Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal (2023). Pythia: A suite for analyzing large language models across training and scaling. [find](https://scholar.google.com/scholar?q=Biderman%2C+S%2E%2C+H%2E+Schoelkopf%2C+Q%2E+Anthony%2C+H%2E+Bradley%2C+K%2E+O%E2%80%99Brien%2C+E%2E+Hallahan%2C+M%2E+A%2E+Khan%2C+S%2E+Purohit%2C+U%2E+S%2E+Prashanth%2C+E%2E+Raff%2C+A%2E+Skowron%2C+L%2E+Sutawika%2C+and+O%2E+van+der+Wal+%282023%29%2E+Pythia%3A+A+suite+for+)
11. Birhane, A., V. U. Prabhu, and E. Kahembwe (2021). Multimodal datasets: misogyny, pornography, and malignant stereotypes. ArXiv abs/2110.01963. — [arXiv:2110.01963](https://arxiv.org/abs/2110.01963) [find](https://scholar.google.com/scholar?q=Birhane%2C+A%2E%2C+V%2E+U%2E+Prabhu%2C+and+E%2E+Kahembwe+%282021%29%2E+Multimodal+datasets%3A+misogyny%2C+pornography%2C+and+malignant+stereotypes%2E+ArXiv+abs%2F2110%2E01963%2E+%E2%80%94+%5BarXiv%3A2110%2E01963%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2110%2E01963%29)
12. Blei, D. M., A. Y. Ng, and M. I. Jordan (2003, mar). Latent dirichlet allocation. J. Mach. Learn. Res. 3 (null), 993–1022. [our card](https://md.research.elamur.ai/free/en/papers/blei-2003-latent-dirichlet-allocation/)
13. Broder, A. (1997). On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No.97TB100171), pp. 21–29. [find](https://scholar.google.com/scholar?q=Broder%2C+A%2E+%281997%29%2E+On+the+resemblance+and+containment+of+documents%2E+In+Proceedings%2E+Compression+and+Complexity+of+SEQUENCES+1997+%28Cat%2E+No%2E97TB100171%29%2C+pp%2E+21%E2%80%9329%2E)
14. Brown, T., B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berne… [our card](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
15. Byeon, M., B. Park, H. Kim, S. Lee, W. Baek, and S. Kim (2022). Coyo-700m: Image-text pair dataset. [https://github.com/kakaobrain/coyo-dataset](https://github.com/kakaobrain/coyo-dataset). [↗](https://github.com/kakaobrain/coyo-dataset)
16. Caswell, I., T. Breiner, D. van Esch, and A. Bapna (2020). Language id in the wild: Unexpected challenges on the path to a thousand-language web text corpus. ArXiv abs/2010.14571. [our card](https://md.research.elamur.ai/free/en/papers/caswell-2020-language-id-wild-unexpected/)
17. Changpinyo, S., P. Sharma, N. Ding, and R. Soricut (2021). Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR. [find](https://scholar.google.com/scholar?q=Changpinyo%2C+S%2E%2C+P%2E+Sharma%2C+N%2E+Ding%2C+and+R%2E+Soricut+%282021%29%2E+Conceptual+12M%3A+Pushing+web%2Dscale+image%2Dtext+pre%2Dtraining+to+recognize+long%2Dtail+visual+concepts%2E+In+CVPR%2E)
18. Chowdhery, A., S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y. Tay, N. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope, J. Bradbury, J. Austin, M. Isard, G. Gur… [our card](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
19. Das, A., S. Kottur, K. Gupta, A. Singh, D. Yadav, J. M. F. Moura, D. Parikh, and D. Batra (2017, July). Visual dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). [find](https://scholar.google.com/scholar?q=Das%2C+A%2E%2C+S%2E+Kottur%2C+K%2E+Gupta%2C+A%2E+Singh%2C+D%2E+Yadav%2C+J%2E+M%2E+F%2E+Moura%2C+D%2E+Parikh%2C+and+D%2E+Batra+%282017%2C+July%29%2E+Visual+dialog%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition+)
20. Dehghani, M., J. Djolonga, B. Mustafa, P. Padlewski, J. Heek, J. Gilmer, A. Steiner, M. Caron, R. Geirhos, I. Alabdulmohsin, R. Jenatton, L. Beyer, M. Tschannen, A. Arnab, X. Wang, C. Riquelme, M. Minderer, J. Puigcerver, U. Evci, M. Kumar, S. van Steenkiste, G. F. Elsayed, A. Mahendran, F. Yu, A. O… [find](https://scholar.google.com/scholar?q=Dehghani%2C+M%2E%2C+J%2E+Djolonga%2C+B%2E+Mustafa%2C+P%2E+Padlewski%2C+J%2E+Heek%2C+J%2E+Gilmer%2C+A%2E+Steiner%2C+M%2E+Caron%2C+R%2E+Geirhos%2C+I%2E+Alabdulmohsin%2C+R%2E+Jenatton%2C+L%2E+Beyer%2C+M%2E+Tschannen%2C+A%2E+Arnab%2C+X%2E+Wang%2C+C%2E+Riquelme%2C+M%2E+Min)
21. Deng, X., P. Shiralkar, C. Lockard, B. Huang, and H. Sun (2022). Dom-lm: Learning generalizable representations for html documents. ArXiv abs/2201.10608. — [arXiv:2201.10608](https://arxiv.org/abs/2201.10608) [find](https://scholar.google.com/scholar?q=Deng%2C+X%2E%2C+P%2E+Shiralkar%2C+C%2E+Lockard%2C+B%2E+Huang%2C+and+H%2E+Sun+%282022%29%2E+Dom%2Dlm%3A+Learning+generalizable+representations+for+html+documents%2E+ArXiv+abs%2F2201%2E10608%2E+%E2%80%94+%5BarXiv%3A2201%2E10608%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F220)
22. Desai, K., G. Kaul, Z. Aysola, and J. Johnson (2021). Redcaps: Web-curated image-text data created by the people, for the people. In J. Vanschoren and S. Yeung (Eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, Volume 1. Curran. [find](https://scholar.google.com/scholar?q=Desai%2C+K%2E%2C+G%2E+Kaul%2C+Z%2E+Aysola%2C+and+J%2E+Johnson+%282021%29%2E+Redcaps%3A+Web%2Dcurated+image%2Dtext+data+created+by+the+people%2C+for+the+people%2E+In+J%2E+Vanschoren+and+S%2E+Yeung+%28Eds%2E%29%2C+Proceedings+of+the+Neural+Inform)
23. Dodge, J., A. Marasović, G. Ilharco, D. Groeneveld, M. Mitchell, and M. Gardner (2021). Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Dodge%2C+J%2E%2C+A%2E+Marasovi%C4%87%2C+G%2E+Ilharco%2C+D%2E+Groeneveld%2C+M%2E+Mitchell%2C+and+M%2E+Gardner+%282021%29%2E+Documenting+large+webtext+corpora%3A+A+case+study+on+the+colossal+clean+crawled+corpus%2E+In+Conference+on+Empirical)
24. Gadre, S. Y., G. Ilharco, A. Fang, J. Hayase, G. Smyrnis, T. Nguyen, R. Marten, M. Wortsman, D. Ghosh, J. Zhang, E. Orgad, R. Entezari, G. Daras, S. Pratt, V. Ramanujan, Y. Bitton, K. Marathe, S. Mussmann, R. Vencu, M. Cherti, R. Krishna, P. W. Koh, O. Saukh, A. Ratner, S. Song, H. Hajishirzi, A. Fa… [find](https://scholar.google.com/scholar?q=Gadre%2C+S%2E+Y%2E%2C+G%2E+Ilharco%2C+A%2E+Fang%2C+J%2E+Hayase%2C+G%2E+Smyrnis%2C+T%2E+Nguyen%2C+R%2E+Marten%2C+M%2E+Wortsman%2C+D%2E+Ghosh%2C+J%2E+Zhang%2C+E%2E+Orgad%2C+R%2E+Entezari%2C+G%2E+Daras%2C+S%2E+Pratt%2C+V%2E+Ramanujan%2C+Y%2E+Bitton%2C+K%2E+Marathe%2C+S%2E+Muss)
25. Gao, L., S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy (2020). The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 . [find](https://scholar.google.com/scholar?q=Gao%2C+L%2E%2C+S%2E+Biderman%2C+S%2E+Black%2C+L%2E+Golding%2C+T%2E+Hoppe%2C+C%2E+Foster%2C+J%2E+Phang%2C+H%2E+He%2C+A%2E+Thite%2C+N%2E+Nabeshima%2C+S%2E+Presser%2C+and+C%2E+Leahy+%282020%29%2E+The+Pile%3A+An+800gb+dataset+of+diverse+text+for+language+model)
26. Gokaslan, A. and V. Cohen (2019). Openwebtext corpus. [http://Skylion007.github.io/](http://Skylion007.github.io/) OpenWebTextCorpus. [↗](http://Skylion007.github.io/)
27. Gu, J., X. Meng, G. Lu, L. Hou, N. Minzhe, X. Liang, L. Yao, R. Huang, W. Zhang, X. Jiang, C. XU, and H. Xu (2022). Wukong: A 100 million large-scale chinese cross-modal pre-training benchmark. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Advances in Neural Informatio… [find](https://scholar.google.com/scholar?q=Gu%2C+J%2E%2C+X%2E+Meng%2C+G%2E+Lu%2C+L%2E+Hou%2C+N%2E+Minzhe%2C+X%2E+Liang%2C+L%2E+Yao%2C+R%2E+Huang%2C+W%2E+Zhang%2C+X%2E+Jiang%2C+C%2E+XU%2C+and+H%2E+Xu+%282022%29%2E+Wukong%3A+A+100+million+large%2Dscale+chinese+cross%2Dmodal+pre%2Dtraining+benchmark%2E+In+S%2E+)
28. Gurari, D., Q. Li, A. J. Stangl, A. Guo, C. Lin, K. Grauman, J. Luo, and J. P. Bigham (2018). Vizwiz grand challenge: Answering visual questions from blind people. [find](https://scholar.google.com/scholar?q=Gurari%2C+D%2E%2C+Q%2E+Li%2C+A%2E+J%2E+Stangl%2C+A%2E+Guo%2C+C%2E+Lin%2C+K%2E+Grauman%2C+J%2E+Luo%2C+and+J%2E+P%2E+Bigham+%282018%29%2E+Vizwiz+grand+challenge%3A+Answering+visual+questions+from+blind+people%2E)
29. Heafield, K. (2011, July). KenLM: Faster and smaller language model queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation, Edinburgh, Scotland, pp. 187– 197. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/heafield-2011-kenlm-faster-smaller-language-model/)
30. Hoffmann, J., S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre (2022). Training com… [our card](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)
31. Huang, S., L. Dong, W. Wang, Y. Hao, S. Singhal, S. Ma, T. Lv, L. Cui, O. K. Mohammed, B. Patra, Q. Liu, K. Aggarwal, Z. Chi, J. Bjorck, V. Chaudhary, S. Som, X. Song, and F. Wei (2023). Language is not all you need: Aligning perception with language models. [our card](https://md.research.elamur.ai/free/en/papers/huang-2023-language-all-need-aligning-perception/)
32. Jaegle, A., F. Gimeno, A. Brock, A. Zisserman, O. Vinyals, and J. Carreira (2021). Perceiver: General perception with iterative attention. [our card](https://md.research.elamur.ai/free/en/papers/jaegle-2021-perceiver-general-perception-iterative-attention/)
33. Jia, C., Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. V. Le, Y.-H. Sung, Z. Li, and T. Duerig (2021). Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning. [our card](https://md.research.elamur.ai/free/en/papers/jia-2021-scaling-up-visual-vision-language/)
34. Jiang, A. Q., S. Welleck, J. P. Zhou, T. Lacroix, J. Liu, W. Li, M. Jamnik, G. Lample, and Y. Wu (2023). Draft, sketch, and prove: Guiding formal theorem provers with informal proofs. In The Eleventh International Conference on Learning Representations. [our card](https://md.research.elamur.ai/free/en/papers/jiang-2022-draft-sketch-prove-guiding-formal/)
35. Joulin, A., E. Grave, P. Bojanowski, and T. Mikolov (2017, April). Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, Valencia, Spain, pp. 427–431. Association for Co… [find](https://scholar.google.com/scholar?q=Joulin%2C+A%2E%2C+E%2E+Grave%2C+P%2E+Bojanowski%2C+and+T%2E+Mikolov+%282017%2C+April%29%2E+Bag+of+tricks+for+efficient+text+classification%2E+In+Proceedings+of+the+15th+Conference+of+the+European+Chapter+of+the+Association+for)
36. Kaplan, J., S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei (2020). Scaling laws for neural language models. [our card](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/)
37. Kärkkäinen, K. and J. Joo (2021). Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), 1547–1557. [find](https://scholar.google.com/scholar?q=K%C3%A4rkk%C3%A4inen%2C+K%2E+and+J%2E+Joo+%282021%29%2E+Fairface%3A+Face+attribute+dataset+for+balanced+race%2C+gender%2C+and+age+for+bias+measurement+and+mitigation%2E+2021+IEEE+Winter+Conference+on+Applications+of+Computer+Visio)
38. Kiela, D., H. Firooz, A. Mohan, V. Goswami, A. Singh, P. Ringshia, and D. Testuggine (2020). The hateful memes challenge: Detecting hate speech in multimodal memes. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (Eds.), Advances in Neural Information Processing Systems, Volume 33, p… [our card](https://md.research.elamur.ai/free/en/papers/kiela-2020-hateful-memes-challenge-detecting-hate/)
39. Koh, J. Y., R. Salakhutdinov, and D. Fried (2023). Grounding language models to images for multimodal generation. [find](https://scholar.google.com/scholar?q=Koh%2C+J%2E+Y%2E%2C+R%2E+Salakhutdinov%2C+and+D%2E+Fried+%282023%29%2E+Grounding+language+models+to+images+for+multimodal+generation%2E)
40. Laborde, G. Deep nn for nsfw detection. Laurençon, H., L. Saulnier, T. Wang, C. Akiki, A. Villanova del Moral, T. Le Scao, L. Von Werra, C. Mou, E. González Ponferrada, H. Nguyen, J. Frohberg, M. Šaško, Q. Lhoest, A. McMillan-Major, G. Dupont, S. Biderman, A. Rogers, L. Ben allal, F. De Toni, G. Pis… [our card](https://md.research.elamur.ai/free/en/papers/lauren-ccon-2023-bigscience-roots-corpus-1-6tb/)
41. Lee, K., D. Ippolito, A. Nystrom, C. Zhang, D. Eck, C. Callison-Burch, and N. Carlini (2022). Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Lee%2C+K%2E%2C+D%2E+Ippolito%2C+A%2E+Nystrom%2C+C%2E+Zhang%2C+D%2E+Eck%2C+C%2E+Callison%2DBurch%2C+and+N%2E+Carlini+%282022%29%2E+Deduplicating+training+data+makes+language+models+better%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the)
42. Li, J., D. Li, S. Savarese, and S. Hoi (2023). Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. [our card](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/)
43. Li, J., D. Li, C. Xiong, and S. Hoi (2022). Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML. [our card](https://md.research.elamur.ai/free/en/papers/li-2022-blip-bootstrapping-language-image-pre/)
44. Li, R., L. B. Allal, Y. Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim, Q. Liu, E. Zheltonozhskii, T. Y. Zhuo, T. Wang, O. Dehaene, M. Davaadorj, J. Lamy-Poirier, J. Monteiro, O. Shliazhko, N. Gontier, N. Meade, A. Zebaze, M.-H. Yee, L. K. Umapathi, J. Zhu, B. Lipkin, M… [find](https://scholar.google.com/scholar?q=Li%2C+R%2E%2C+L%2E+B%2E+Allal%2C+Y%2E+Zi%2C+N%2E+Muennighoff%2C+D%2E+Kocetkov%2C+C%2E+Mou%2C+M%2E+Marone%2C+C%2E+Akiki%2C+J%2E+Li%2C+J%2E+Chim%2C+Q%2E+Liu%2C+E%2E+Zheltonozhskii%2C+T%2E+Y%2E+Zhuo%2C+T%2E+Wang%2C+O%2E+Dehaene%2C+M%2E+Davaadorj%2C+J%2E+Lamy%2DPoirier%2C+J%2E+Mont)
45. Lin, T.-Y., M. Maire, S. Belongie, L. Bourdev, R. Girshick, J. Hays, P. Perona, D. Ramanan, C. L. Zitnick, and P. Dollár (2014). Microsoft coco: Common objects in context. cite arxiv:1405.0312Comment: 1) updated annotation pipeline description and figures; 2) added new section describing datasets sp… [our card](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/)
46. Liu, S., L. Fan, E. Johns, Z. Yu, C. Xiao, and A. Anandkumar (2023). Prismer: A vision-language model with an ensemble of experts. arXiv preprint arXiv:2303.02506 . — [arXiv:2303.02506](https://arxiv.org/abs/2303.02506) [find](https://scholar.google.com/scholar?q=Liu%2C+S%2E%2C+L%2E+Fan%2C+E%2E+Johns%2C+Z%2E+Yu%2C+C%2E+Xiao%2C+and+A%2E+Anandkumar+%282023%29%2E+Prismer%3A+A+vision%2Dlanguage+model+with+an+ensemble+of+experts%2E+arXiv+preprint+arXiv%3A2303%2E02506+%2E+%E2%80%94+%5BarXiv%3A2303%2E02506%5D%28https%3A%2F%2Farxiv%2E)
47. Liu, Y., G. Zhu, B. Zhu, Q. Song, G. Ge, H. Chen, G. Qiao, R. Peng, L. Wu, and J. Wang (2022). Taisu: A 166m large-scale high-quality dataset for chinese vision-language pre-training. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Advances in Neural Information Processi… [find](https://scholar.google.com/scholar?q=Liu%2C+Y%2E%2C+G%2E+Zhu%2C+B%2E+Zhu%2C+Q%2E+Song%2C+G%2E+Ge%2C+H%2E+Chen%2C+G%2E+Qiao%2C+R%2E+Peng%2C+L%2E+Wu%2C+and+J%2E+Wang+%282022%29%2E+Taisu%3A+A+166m+large%2Dscale+high%2Dquality+dataset+for+chinese+vision%2Dlanguage+pre%2Dtraining%2E+In+S%2E+Koyejo%2C+S%2E)
48. Loshchilov, I. and F. Hutter (2017). CoRR abs/1711.05101. [find](https://scholar.google.com/scholar?q=Loshchilov%2C+I%2E+and+F%2E+Hutter+%282017%29%2E+CoRR+abs%2F1711%2E05101%2E)
49. Fixing weight decay regularization in adam. [find](https://scholar.google.com/scholar?q=Fixing+weight+decay+regularization+in+adam%2E)
50. Luccioni, A. S., C. Akiki, M. Mitchell, and Y. Jernite (2023). Stable bias: Analyzing societal representations in diffusion models. [find](https://scholar.google.com/scholar?q=Luccioni%2C+A%2E+S%2E%2C+C%2E+Akiki%2C+M%2E+Mitchell%2C+and+Y%2E+Jernite+%282023%29%2E+Stable+bias%3A+Analyzing+societal+representations+in+diffusion+models%2E)
51. Marino, K., M. Rastegari, A. Farhadi, and R. Mottaghi (2019). Ok-vqa: A visual question answering benchmark requiring external knowledge. In Conference on Computer Vision and Pattern Recognition (CVPR). [find](https://scholar.google.com/scholar?q=Marino%2C+K%2E%2C+M%2E+Rastegari%2C+A%2E+Farhadi%2C+and+R%2E+Mottaghi+%282019%29%2E+Ok%2Dvqa%3A+A+visual+question+answering+benchmark+requiring+external+knowledge%2E+In+Conference+on+Computer+Vision+and+Pattern+Recognition+%28CVPR)
52. Mishra, A., K. Alahari, and C. V. Jawahar (2012). Scene text recognition using higher order language priors. In BMVC. [find](https://scholar.google.com/scholar?q=Mishra%2C+A%2E%2C+K%2E+Alahari%2C+and+C%2E+V%2E+Jawahar+%282012%29%2E+Scene+text+recognition+using+higher+order+language+priors%2E+In+BMVC%2E)
53. Nichol, A., P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen (2022). Glide: Towards photorealistic image generation and editing with text-guided diffusion models. [find](https://scholar.google.com/scholar?q=Nichol%2C+A%2E%2C+P%2E+Dhariwal%2C+A%2E+Ramesh%2C+P%2E+Shyam%2C+P%2E+Mishkin%2C+B%2E+McGrew%2C+I%2E+Sutskever%2C+and+M%2E+Chen+%282022%29%2E+Glide%3A+Towards+photorealistic+image+generation+and+editing+with+text%2Dguided+diffusion+models%2E)
54. Ortiz Suárez, P. J., L. Romary, and B. Sagot (2020, July). A monolingual approach to contextualized word embeddings for mid-resource languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 1703–1714. Association for Computational Linguistics… [our card](https://md.research.elamur.ai/free/en/papers/suarez-2020-monolingual-approach-contextualized-word-embeddings/)
55. Ouyang, L., J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe (2022). Training language models to follow instructions with human feed… [our card](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
56. Piktus, A., C. Akiki, P. Villegas, H. Laurençon, G. Dupont, A. S. Luccioni, Y. Jernite, and A. Rogers (2023). The roots search tool: Data transparency for llms. [find](https://scholar.google.com/scholar?q=Piktus%2C+A%2E%2C+C%2E+Akiki%2C+P%2E+Villegas%2C+H%2E+Lauren%C3%A7on%2C+G%2E+Dupont%2C+A%2E+S%2E+Luccioni%2C+Y%2E+Jernite%2C+and+A%2E+Rogers+%282023%29%2E+The+roots+search+tool%3A+Data+transparency+for+llms%2E)
57. Radenovic, F., A. Dubey, A. Kadian, T. Mihaylov, S. Vandenhende, Y. Patel, Y. Wen, V. Ramanathan, and D. Mahajan (2023). Filtering, distillation, and hard negatives for vision-language pre-training. [find](https://scholar.google.com/scholar?q=Radenovic%2C+F%2E%2C+A%2E+Dubey%2C+A%2E+Kadian%2C+T%2E+Mihaylov%2C+S%2E+Vandenhende%2C+Y%2E+Patel%2C+Y%2E+Wen%2C+V%2E+Ramanathan%2C+and+D%2E+Mahajan+%282023%29%2E+Filtering%2C+distillation%2C+and+hard+negatives+for+vision%2Dlanguage+pre%2Dtraining%2E)
58. Radford, A., J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever (2021). Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. [our card](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/)
59. Rae, J. W., S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van den Driessche, L. A. Hendricks, M. Rauh, P.-S. Huang, A. Glaese, J. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Me… [find](https://scholar.google.com/scholar?q=Rae%2C+J%2E+W%2E%2C+S%2E+Borgeaud%2C+T%2E+Cai%2C+K%2E+Millican%2C+J%2E+Hoffmann%2C+F%2E+Song%2C+J%2E+Aslanides%2C+S%2E+Henderson%2C+R%2E+Ring%2C+S%2E+Young%2C+E%2E+Rutherford%2C+T%2E+Hennigan%2C+J%2E+Menick%2C+A%2E+Cassirer%2C+R%2E+Powell%2C+G%2E+van+den+Driessche%2C+)
60. Raffel, C., N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu (2019). Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints. [our card](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/)
61. Ramesh, A., P. Dhariwal, A. Nichol, C. Chu, and M. Chen (2022). Hierarchical text-conditional image generation with clip latents. [our card](https://md.research.elamur.ai/free/en/papers/ramesh-2022-hierarchical-text-conditional-image-generation/)
62. Rombach, R., A. Blattmann, D. Lorenz, P. Esser, and B. Ommer (2021). High-resolution image synthesis with latent diffusion models. [find](https://scholar.google.com/scholar?q=Rombach%2C+R%2E%2C+A%2E+Blattmann%2C+D%2E+Lorenz%2C+P%2E+Esser%2C+and+B%2E+Ommer+%282021%29%2E+High%2Dresolution+image+synthesis+with+latent+diffusion+models%2E)
63. Saharia, C., W. Chan, S. Saxena, L. Li, J. Whang, E. Denton, S. K. S. Ghasemipour, B. K. Ayan, S. S. Mahdavi, R. G. Lopes, T. Salimans, J. Ho, D. J. Fleet, and M. Norouzi (2022). Photorealistic text-to-image diffusion models with deep language understanding. [our card](https://md.research.elamur.ai/free/en/papers/saharia-2022-photorealistic-text-image-diffusion-models/)
64. Schuhmann, C., R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, P. Schramowski, S. Kundurthy, K. Crowson, L. Schmidt, R. Kaczmarczyk, and J. Jitsev (2022). Laion-5b: An open large-scale dataset for training next generation image-text models. In… [our card](https://md.research.elamur.ai/free/en/papers/schuhmann-2022-laion-5b-open-large-scale/)
65. Schuhmann, C., R. Vencu, R. Beaumont, R. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki (2021). Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. [our card](https://md.research.elamur.ai/free/en/papers/schuhmann-2021-laion-400m-open-dataset-clip/)
66. Sharma, P., N. Ding, S. Goodman, and R. Soricut (2018). Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of ACL. [our card](https://md.research.elamur.ai/free/en/papers/sharma-2018-conceptual-captions-cleaned-hypernymed-image/)
67. Singh, A., R. Hu, V. Goswami, G. Couairon, W. Galuba, M. Rohrbach, and D. Kiela (2022). FLAVA: A foundational language and vision alignment model. In CVPR. [find](https://scholar.google.com/scholar?q=Singh%2C+A%2E%2C+R%2E+Hu%2C+V%2E+Goswami%2C+G%2E+Couairon%2C+W%2E+Galuba%2C+M%2E+Rohrbach%2C+and+D%2E+Kiela+%282022%29%2E+FLAVA%3A+A+foundational+language+and+vision+alignment+model%2E+In+CVPR%2E)
68. Singh, A., V. Natarjan, M. Shah, Y. Jiang, X. Chen, D. Parikh, and M. Rohrbach (2019). Towards vqa models that can read. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8317–8326. [find](https://scholar.google.com/scholar?q=Singh%2C+A%2E%2C+V%2E+Natarjan%2C+M%2E+Shah%2C+Y%2E+Jiang%2C+X%2E+Chen%2C+D%2E+Parikh%2C+and+M%2E+Rohrbach+%282019%29%2E+Towards+vqa+models+that+can+read%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognitio)
69. Sorscher, B., R. Geirhos, S. Shekhar, S. Ganguli, and A. Morcos (2022). Beyond neural scaling laws: beating power law scaling via data pruning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Advances in Neural Information Processing Systems, Volume 35, pp. 19523–19536.… [our card](https://md.research.elamur.ai/free/en/papers/sorscher-2022-neural-scaling-laws-beating-power/)
70. Srinivasan, K., K. Raman, J. Chen, M. Bendersky, and M. Najork (2021). Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’21, New York, NY, US… [find](https://scholar.google.com/scholar?q=Srinivasan%2C+K%2E%2C+K%2E+Raman%2C+J%2E+Chen%2C+M%2E+Bendersky%2C+and+M%2E+Najork+%282021%29%2E+Wit%3A+Wikipedia%2Dbased+image+text+dataset+for+multimodal+multilingual+machine+learning%2E+In+Proceedings+of+the+44th+International+AC)
71. Team, M. N. (2023). Introducing mpt-7b: A new standard for open-source, commercially usable llms. [find](https://scholar.google.com/scholar?q=Team%2C+M%2E+N%2E+%282023%29%2E+Introducing+mpt%2D7b%3A+A+new+standard+for+open%2Dsource%2C+commercially+usable+llms%2E)
72. Touvron, H., T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample (2023). Llama: Open and efficient foundation language models. [our card](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/)
73. Wang, P., A. Yang, R. Men, J. Lin, S. Bai, Z. Li, J. Ma, C. Zhou, J. Zhou, and H. Yang (2022, 17–23 Jul). OFA: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato (Eds.), Pr… [our card](https://md.research.elamur.ai/free/en/papers/wang-2022-ofa-unifying-architectures-tasks-modalities/)
74. Wang, Q., Y. Fang, A. Ravula, F. Feng, X. Quan, and D. Liu (2022). Webformer: The web-page transformer for structure information extraction. [find](https://scholar.google.com/scholar?q=Wang%2C+Q%2E%2C+Y%2E+Fang%2C+A%2E+Ravula%2C+F%2E+Feng%2C+X%2E+Quan%2C+and+D%2E+Liu+%282022%29%2E+Webformer%3A+The+web%2Dpage+transformer+for+structure+information+extraction%2E)
75. Wang, W., H. Bao, L. Dong, J. Bjorck, Z. Peng, Q. Liu, K. Aggarwal, O. K. Mohammed, S. Singhal, S. Som, and F. Wei (2022). Image as a foreign language: Beit pretraining for all vision and vision-language tasks. [find](https://scholar.google.com/scholar?q=Wang%2C+W%2E%2C+H%2E+Bao%2C+L%2E+Dong%2C+J%2E+Bjorck%2C+Z%2E+Peng%2C+Q%2E+Liu%2C+K%2E+Aggarwal%2C+O%2E+K%2E+Mohammed%2C+S%2E+Singhal%2C+S%2E+Som%2C+and+F%2E+Wei+%282022%29%2E+Image+as+a+foreign+language%3A+Beit+pretraining+for+all+vision+and+vision%2Dlangu)
76. Webster, R., J. Rabin, L. Simon, and F. Jurie (2023). On the de-duplication of laion-2b. [find](https://scholar.google.com/scholar?q=Webster%2C+R%2E%2C+J%2E+Rabin%2C+L%2E+Simon%2C+and+F%2E+Jurie+%282023%29%2E+On+the+de%2Dduplication+of+laion%2D2b%2E)
77. Workshop, B., :, T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilić, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, J. Tow, A. M. Rush, S. Biderman, A. Webson, P. S. Ammanamanchi, T. Wang, B. Sagot, N. Muennighoff, A. V. del Moral, O. Ruwase, R. Bawden, S. Bekman, A. McMillan-Major, I. B… [find](https://scholar.google.com/scholar?q=Workshop%2C+B%2E%2C+%3A%2C+T%2E+L%2E+Scao%2C+A%2E+Fan%2C+C%2E+Akiki%2C+E%2E+Pavlick%2C+S%2E+Ili%C4%87%2C+D%2E+Hesslow%2C+R%2E+Castagn%C3%A9%2C+A%2E+S%2E+Luccioni%2C+F%2E+Yvon%2C+M%2E+Gall%C3%A9%2C+J%2E+Tow%2C+A%2E+M%2E+Rush%2C+S%2E+Biderman%2C+A%2E+Webson%2C+P%2E+S%2E+Ammanamanchi%2C+T%2E+Wang%2C)
78. Xie, S. M., H. Pham, X. Dong, N. Du, H. Liu, Y. Lu, P. Liang, Q. V. Le, T. Ma, and A. W. Yu (2023). Doremi: Optimizing data mixtures speeds up language model pretraining. [find](https://scholar.google.com/scholar?q=Xie%2C+S%2E+M%2E%2C+H%2E+Pham%2C+X%2E+Dong%2C+N%2E+Du%2C+H%2E+Liu%2C+Y%2E+Lu%2C+P%2E+Liang%2C+Q%2E+V%2E+Le%2C+T%2E+Ma%2C+and+A%2E+W%2E+Yu+%282023%29%2E+Doremi%3A+Optimizing+data+mixtures+speeds+up+language+model+pretraining%2E)
79. Yang, Z., Z. Gan, J. Wang, X. Hu, Y. Lu, Z. Liu, and L. Wang (2022). An empirical study of gpt-3 for few-shot knowledge-based vqa. In Proceedings of the AAAI Conference on Artificial Intelligence, Volume 36, pp. 3081–3089. [find](https://scholar.google.com/scholar?q=Yang%2C+Z%2E%2C+Z%2E+Gan%2C+J%2E+Wang%2C+X%2E+Hu%2C+Y%2E+Lu%2C+Z%2E+Liu%2C+and+L%2E+Wang+%282022%29%2E+An+empirical+study+of+gpt%2D3+for+few%2Dshot+knowledge%2Dbased+vqa%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+Vol)
80. Young, P., A. Lai, M. Hodosh, and J. Hockenmaier (2014). From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics 2, 67–78. [find](https://scholar.google.com/scholar?q=Young%2C+P%2E%2C+A%2E+Lai%2C+M%2E+Hodosh%2C+and+J%2E+Hockenmaier+%282014%29%2E+From+image+descriptions+to+visual+denotations%3A+New+similarity+metrics+for+semantic+inference+over+event+descriptions%2E+Transactions+of+the+Assoc)
81. Yu, J., Z. Wang, V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu (2022). Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research. [our card](https://md.research.elamur.ai/free/en/papers/yu-2022-coca-contrastive-captioners-image-text/)
82. Yuan, S., S. Zhao, J. Leng, Z. Xue, H. Zhao, P. Liu, Z. Gong, W. X. Zhao, J. Li, and J. Tang (2022). Wudaomm: A large-scale multi-modal dataset for pre-training models. [find](https://scholar.google.com/scholar?q=Yuan%2C+S%2E%2C+S%2E+Zhao%2C+J%2E+Leng%2C+Z%2E+Xue%2C+H%2E+Zhao%2C+P%2E+Liu%2C+Z%2E+Gong%2C+W%2E+X%2E+Zhao%2C+J%2E+Li%2C+and+J%2E+Tang+%282022%29%2E+Wudaomm%3A+A+large%2Dscale+multi%2Dmodal+dataset+for+pre%2Dtraining+models%2E)
83. Yuksekgonul, M., F. Bianchi, P. Kalluri, D. Jurafsky, and J. Zou (2023). When and why vision-language models behave like bags-of-words, and what to do about it? In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Yuksekgonul%2C+M%2E%2C+F%2E+Bianchi%2C+P%2E+Kalluri%2C+D%2E+Jurafsky%2C+and+J%2E+Zou+%282023%29%2E+When+and+why+vision%2Dlanguage+models+behave+like+bags%2Dof%2Dwords%2C+and+what+to+do+about+it%3F+In+International+Conference+on+Learning)
84. Zhang, B. and R. Sennrich (2019). Root Mean Square Layer Normalization. In Advances in Neural Information Processing Systems 32, Vancouver, Canada. [find](https://scholar.google.com/scholar?q=Zhang%2C+B%2E+and+R%2E+Sennrich+%282019%29%2E+Root+Mean+Square+Layer+Normalization%2E+In+Advances+in+Neural+Information+Processing+Systems+32%2C+Vancouver%2C+Canada%2E)
85. Zhang, J., Y. Zhao, M. Saleh, and P. J. Liu (2019). Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. [our card](https://md.research.elamur.ai/free/en/papers/zhang-2019-pegasus-pre-training-extracted-gap/)
86. Zhang, R., J. Han, A. Zhou, X. Hu, S. Yan, P. Lu, H. Li, P. Gao, and Y. Qiao (2023). Llama-adapter: Efficient fine-tuning of language models with zero-init attention. [our card](https://md.research.elamur.ai/free/en/papers/zhang-2023-llama-adapter-efficient-fine-tuning/)
87. Zhang, S., S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer (2022). Opt: Open pre-trained transformer language models. [our card](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/)
88. Zhou, Y., Y. Sheng, N. H. Vo, N. Edmonds, and S. Tata (2021). Simplified dom trees for transferable attribute extraction from the web. ArXiv abs/2101.02415. — [arXiv:2101.02415](https://arxiv.org/abs/2101.02415) [find](https://scholar.google.com/scholar?q=Zhou%2C+Y%2E%2C+Y%2E+Sheng%2C+N%2E+H%2E+Vo%2C+N%2E+Edmonds%2C+and+S%2E+Tata+%282021%29%2E+Simplified+dom+trees+for+transferable+attribute+extraction+from+the+web%2E+ArXiv+abs%2F2101%2E02415%2E+%E2%80%94+%5BarXiv%3A2101%2E02415%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F)
89. Zhu, W., J. Hessel, A. Awadalla, S. Y. Gadre, J. Dodge, A. Fang, Y. Yu, L. Schmidt, W. Y. Wang, and Y. Choi (2023). Multimodal C4: An open, billion-scale corpus of images interleaved with text. arXiv preprint arXiv:2304.06939 . — [arXiv:2304.06939](https://arxiv.org/abs/2304.06939) [find](https://scholar.google.com/scholar?q=Zhu%2C+W%2E%2C+J%2E+Hessel%2C+A%2E+Awadalla%2C+S%2E+Y%2E+Gadre%2C+J%2E+Dodge%2C+A%2E+Fang%2C+Y%2E+Yu%2C+L%2E+Schmidt%2C+W%2E+Y%2E+Wang%2C+and+Y%2E+Choi+%282023%29%2E+Multimodal+C4%3A+An+open%2C+billion%2Dscale+corpus+of+images+interleaved+with+text%2E+arXiv+)