---
title: "TATL at W-NUT 2020 Task 2 — paper by Nguyen (2020)"
description: "The paper describes a system for W-NUT 2020 Shared Task 2, in which English COVID-19 tweets are labelled as… 0.90 F1. 22 references."
authors: ["Anh Tuan Nguyen"]
year: 2020
venue: "arXiv:2008.12854"
source: "https://arxiv.org/abs/2008.12854"
doi: "10.48550/arxiv.2008.12854"
arxiv: "2008.12854"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/nguyen-2020-tatl-w-nut-2020-task/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nguyen-2020-tatl-w-nut-2020-task/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/9/) / TATL at W-NUT 2020 Task 2: A Transformer-based Baseline System for Identification of Informative COVID-19 English Tweets

# TATL at W-NUT 2020 Task 2: A Transformer-based Baseline System for Identification of Informative COVID-19 English Tweets

**Authors:** [Anh Tuan Nguyen](https://md.research.elamur.ai/free/en/authors/anh-tuan-nguyen/) · **Year:** 2020 · **Venue:** arXiv:2008.12854 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2008.12854) · [PDF](https://arxiv.org/pdf/2008.12854) · [DOI 10.48550/arxiv.2008.12854](https://doi.org/10.48550/arxiv.2008.12854) · [arXiv 2008.12854](https://arxiv.org/abs/2008.12854)

## Summary
The paper describes a system for W-NUT 2020 Shared Task 2, in which English COVID-19 tweets are labelled as informative or uninformative. A tweet counts as informative when it reports on cases (confirmed, suspected, recovered, deaths) or on their location and history. Tweets are short and use informal language, which makes them hard for tools built on edited prose. The organisers released about 10K labelled tweets. The author treats the task as binary classification. A pretrained Transformer encodes each tweet, and a single linear layer with a sigmoid reads the first special token. Several encoders were fine-tuned: RoBERTa, XLM-RoBERTa, BERTweet and ELECTRA, in base and large sizes where available. Their outputs were then combined by unweighted probability averaging or by majority voting. RoBERTa-large was the best single model on validation, and both ensembles beat every individual model. On the hidden test set the voting ensemble scored best, at about 0.90 F1, placing 8th of 56 teams. The paper shows how far a plain fine-tuning baseline can go on a noisy social-media classification task.

## Key points
- Data split, informative/uninformative: training 3303/3697, validation 472/528, test 944/1056 (roughly 10K tweets in total).
- Fine-tuning setup: transformers library, AdamW, batch size 32, learning rates from {1e-5, 2e-5, 5e-5}, 30 epochs, with the checkpoint chosen on validation F1.
- Validation F1 by model: RoBERTa-large 0.918, ELECTRA-large 0.914, RoBERTa-base 0.911, BERTweet 0.909, ELECTRA-base 0.907, XLM-RoBERTa large/base 0.906/0.905.
- Ensembles on validation: averaging 0.927, voting 0.922. On test the order flipped, with voting at 0.9008 and averaging at 0.8988.
- Unexpected results: the tweet-specific BERTweet did not beat RoBERTa-base, and the multilingual XLM-RoBERTa trailed the English-only models.

## Interesting ideas
- Combining several differently pretrained encoders, by averaging probabilities or by counting hard votes, gave a reliable gain over the strongest single model.
- The averaging ensemble won on validation, but the voting ensemble won on the hidden test set, so small dev margins between combination rules may not hold.
- A model pretrained on in-domain tweets did not beat a general model trained the same way, so a domain label on an encoder is no guarantee of improvement.
- A multilingual encoder lagged behind monolingual ones on an English-only task, which suggests that spreading capacity across languages can cost single-language accuracy.
- The task defines informativeness through concrete reportable facts (case counts, status, place, history) rather than through topic alone.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): The same two schemes could combine witness parsers when building silver data, with soft agreement and hard majority agreement tracked side by side.
- **I2** (learn): Evaluation should keep a held-out split that is never used for selection, and should report how rankings shift between dev and test rather than trusting a single dev winner.
- **I3** (tree): When picking an encoder for gparse, compare general and domain-specific checkpoints empirically on the target treebank rather than assuming the specialised one wins.
- **I4** (tree): Since Mova is English-focused, gparse should default to English-only pretrained encoders and use multilingual ones only where cross-lingual transfer is actually needed.
- **I5** (world): A text could be judged informative when its world model gains new entities, quantities or states, giving an explainable filter instead of an opaque classifier score.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (2): [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 1,373 works

## Cite

Cite as: Anh Tuan Nguyen. 2020. TATL at W-NUT 2020 Task 2: A Transformer-based Baseline System for Identification of Informative COVID-19 English Tweets. arXiv:2008.12854.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{nguyen2020tatl,
  title = {{TATL at W-NUT 2020 Task 2: A Transformer-based Baseline System for Identification of Informative COVID-19 English Tweets}},
  author = {Anh Tuan Nguyen},
  year = {2020},
  doi = {10.48550/arxiv.2008.12854},
  eprint = {2008.12854},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2008.12854}
}
```

- [← previous Studies in Rabbinic Hebrew](https://md.research.elamur.ai/free/en/papers/heijmans-2020-studies-rabbinic-hebrew/)
- [next → The Biblical Hebrew infinitive](https://md.research.elamur.ai/free/en/papers/doron-2020-biblical-hebrew-infinitive/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/9/)

## References

22 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: Pretrained language model for scientific text. In EMNLP. [find](https://scholar.google.com/scholar?q=Iz+Beltagy%2C+Kyle+Lo%2C+and+Arman+Cohan%2E+2019%2E+Scibert%3A+Pretrained+language+model+for+scientific+text%2E+In+EMNLP%2E)
2. Qian Chen, Zhu Zhuo, and W. Wang. 2019. Bert for joint intent classification and slot filling. ArXiv, abs/1902.10909. [↗](https://arxiv.org/abs/1902.10909)
3. K. Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. ArXiv, abs/2003.10555. [↗](https://arxiv.org/abs/2003.10555)
4. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.acl-main.747)
5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of NAACL, pages 4171– 4186.
6. Lifu Huang, Heng Ji, Kyunghyun Cho, Ido Dagan, Sebastian Riedel, and Clare Voss. 2018. [Zero-shot transfer learning for event extraction](https://md.research.elamur.ai/free/en/papers/huang-2018-zero-shot-transfer-learning-event/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2160–2170, Melbourne, Australia. Association for Computational Linguistics.
7. Tuan Lai, Trung Bui, Nedim Lipka, and Sheng Li. 2018. Supervised transfer learning for product information question answering. In 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), pages 1109–1114. IEEE. [find](https://scholar.google.com/scholar?q=Tuan+Lai%2C+Trung+Bui%2C+Nedim+Lipka%2C+and+Sheng+Li%2E+2018%2E+Supervised+transfer+learning+for+product+information+question+answering%2E+In+2018+17th+IEEE+International+Conference+on+Machine+Learning+and+Applic)
8. Tuan Lai, Quan Hung Tran, Trung Bui, and Daisuke Kihara. 2019. A gated self-attention memory network for answer selection. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5953–5959, Hong Kong, China. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1909.09696)
9. Tuan Manh Lai, Quan Hung Tran, Trung Bui, and Daisuke Kihara. 2020. A simple but effective bert model for dialog state tracking on resource-limited systems. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8034–8038. IEEE. [↗](https://doi.org/10.48550/arxiv.1910.12995)
10. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. Advances in Neural Information Processing Systems (NeurIPS). [find](https://scholar.google.com/scholar?q=Guillaume+Lample+and+Alexis+Conneau%2E+2019%2E+Cross%2Dlingual+language+model+pretraining%2E+Advances+in+Neural+Information+Processing+Systems+%28NeurIPS%29%2E)
11. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://md.research.elamur.ai/free/en/papers/lee-2019-biobert-pre-trained-biomedical-language/). Bioinformatics. [↗](https://doi.org/10.1093/bioinformatics/btz682)
12. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint, arXiv:1907.11692. [↗](https://arxiv.org/abs/1907.11692)
13. Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2019%2E+Decoupled+Weight+Decay+Regularization%2E+In+Proceedings+of+ICLR%2E)
14. Sugeerth Murugesan, Sana Malik, Fan Du, Eunyee Koh, and Tuan Manh Lai. 2019. Deepcompare: Visual and interactive comparison of deep learning model performance. IEEE computer graphics and applications, 39(5):47–59. [↗](https://doi.org/10.1109/mcg.2019.2919033)
15. Dat Quoc Nguyen and Anh Tuan Nguyen. 2020. [PhoBERT: Pre-trained language models for Vietnamese](https://md.research.elamur.ai/free/en/papers/nguyen-2020-phobert-pre-trained-language-models/). arXiv preprint, arXiv:2003.00744. [↗](https://arxiv.org/abs/2003.00744)
16. Dat Quoc Nguyen, Thanh Vu, and A. Nguyen. 2020a. [Bertweet: A pre-trained language model for english tweets](https://md.research.elamur.ai/free/en/papers/nguyen-2020-bertweet-pre-trained-language-model/). ArXiv, abs/2005.10200. [↗](https://arxiv.org/abs/2005.10200)
17. Dat Quoc Nguyen, Thanh Vu, Afshin Rahimi, Mai Hoang Dao, Linh The Nguyen, and Long Doan. 2020b. WNUT-2020 Task 2: Identification of Informative COVID-19 English Tweets. In Proceedings of the 6th Workshop on Noisy User-generated Text. [find](https://scholar.google.com/scholar?q=Dat+Quoc+Nguyen%2C+Thanh+Vu%2C+Afshin+Rahimi%2C+Mai+Hoang+Dao%2C+Linh+The+Nguyen%2C+and+Long+Doan%2E+2020b%2E+WNUT%2D2020+Task+2%3A+Identification+of+Informative+COVID%2D19+English+Tweets%2E+In+Proceedings+of+the+6th+Works)
18. Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. [Cross-lingual name tagging and linking for 282 languages](https://md.research.elamur.ai/free/en/papers/pan-2017-cross-lingual-name-tagging-linking/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946–1958, Vancouver, Canada. Association for Computational Linguistics.
19. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. [↗](https://doi.org/10.18653/v1/w18-5446)
20. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. HuggingFace’s [Transformers: State-of-the-art Natural Language Processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). arXiv preprint, arXiv:1910.03771. [↗](https://arxiv.org/abs/1910.03771)
21. Qizhe Xie, E. Hovy, Minh-Thang Luong, and Quoc V. Le. 2020. Self-training with noisy student improves imagenet classification. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695. [find](https://scholar.google.com/scholar?q=Qizhe+Xie%2C+E%2E+Hovy%2C+Minh%2DThang+Luong%2C+and+Quoc+V%2E+Le%2E+2020%2E+Self%2Dtraining+with+noisy+student+improves+imagenet+classification%2E+2020+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition+%28CVPR)
22. Seunghyun Yoon, Franck Dernoncourt, Doo Soon Kim, Trung Bui, and Kyomin Jung. 2019. A compare-aggregate model with latent clustering for answer selection. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pages 2093–2096. [find](https://scholar.google.com/scholar?q=Seunghyun+Yoon%2C+Franck+Dernoncourt%2C+Doo+Soon+Kim%2C+Trung+Bui%2C+and+Kyomin+Jung%2E+2019%2E+A+compare%2Daggregate+model+with+latent+clustering+for+answer+selection%2E+In+Proceedings+of+the+28th+ACM+International+)

*Updated 24 October 2026*