---
title: "Not-NUTs at WNUT-2020 Task 2 — paper by Hoang and Vu (2020)"
description: "The authors use the shared-task corpus of 10,000 tweets and build on BERTweet, a Transformer pre-trained on tweets, with a… 12 references."
authors: ["Thai Hoang", "Phuong Vu"]
year: 2020
venue: "Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020)"
source: "https://aclanthology.org/2020.wnut-1.69/"
doi: "10.18653/v1/2020.wnut-1.69"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/hoang-2020-not-nuts-wnut-2020-task/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/hoang-2020-not-nuts-wnut-2020-task/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/11/) / Not-NUTs at WNUT-2020 Task 2: A BERT-based System in Identifying Informative COVID-19 English Tweets

# Not-NUTs at WNUT-2020 Task 2: A BERT-based System in Identifying Informative COVID-19 English Tweets

**Authors:** [Thai Hoang](https://md.research.elamur.ai/free/en/authors/thai-hoang/) and Phuong Vu · **Year:** 2020 · **Venue:** [Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020)](https://md.research.elamur.ai/free/en/venues/wnut/) ([Workshop on Noisy User-generated Text 2020](https://md.research.elamur.ai/free/en/venues/wnut/2020/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.wnut-1.69/) · [PDF](https://aclanthology.org/2020.wnut-1.69.pdf) · [DOI 10.18653/v1/2020.wnut-1.69](https://doi.org/10.18653/v1/2020.wnut-1.69) · [OpenAlex](https://openalex.org/W3086597931)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
The paper tackles WNUT-2020 Task 2: deciding whether an English COVID-19 tweet is informative, meaning it reports suspected, confirmed, recovered or fatal cases, or the location and travel history of those cases. The authors use the shared-task corpus of 10,000 tweets and build on BERTweet, a Transformer pre-trained on tweets, with a small stack of linear layers as the classification head. They compare several ways of pooling the [CLS] representation across Transformer layers. They also introduce a "Global Local BERTweet" that trains three encoders together: one reads the whole tweet, one the first part and one the rest. Training runs in two stages, first fitting only the head and then fine-tuning the full network. Many configurations are then combined by majority voting or by averaging softmax outputs. The submitted ensemble reached an F1 of 0.8991 on the hidden test set, placing 12th of 56 teams and about one point behind the winner. The work shows that cheap layer-mixing, part-of-text encoders and ensembling add measurable gains over a single fine-tuned domain model.

## Key points
- The data is 10,000 tweets, each labelled by 3 annotators, with a Fleiss' Kappa of 0.818. It is split 70/10/20 into train, validation and test. For the final run, train and validation were merged and re-split 90/10.
- tf-idf baselines built with sklearn scored F1 0.7827 for Logistic Regression, 0.7678 for SVM and 0.7486 for Naive Bayes on validation.
- Using only BERTweet's last layer gave F1 0.8912. Concatenating the last and first layers gave the best single-encoder result at 0.9045. Global Local variants reached up to 0.9075.
- On validation, majority voting scored F1 0.9130 and softmax averaging 0.9111. The test result was 0.8991, against 0.9096 for the top team.
- After the competition, the authors grouped tweets by length: short (0–22 words), medium (23–44) and long (>44). For each group they picked the 7 models with the most correct training predictions. This raised validation F1 to 0.9182.

## Interesting ideas
- The text is encoded both as a whole and as separate segments, and the pooled vectors are concatenated, so that locally concentrated cues are not diluted by the rest of the sequence.
- Representations from shallow and deep layers are mixed (for example first + last) instead of relying only on the top layer. This helped because lower layers keep surface and lexical signals that the top layer partly loses.
- Training runs in two stages: the pre-trained encoder is frozen while a fresh head is fitted, then everything is unfrozen at a lower learning rate. This keeps the head's random start from damaging the pre-trained weights.
- Ensemble members are chosen per input-length bucket, keeping the models that perform best on that range of lengths. This treats input length as a factor that decides which model to trust.
- Cheap tf-idf baselines are reported next to the neural system, which shows how much the expensive model actually adds.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (frames): when assigning speech acts or detecting the newsworthy clause, the analysis could score the main clause and the trailing segments separately, then combine the scores while keeping track of which segment drove the decision.
- **I2** (tree): gparse could feed its arc scorer a concatenation of an early and a late transformer layer, and test on a held-out treebank whether LAS improves.
- **I3** (learn): when new heads such as role or frame classifiers are attached to a pretrained parser encoder, warming up the head first should make training on silver data more stable.
- **I4** (learn): evaluation could report parser and reader accuracy broken down by sentence length. A length-aware choice among witness parsers could then decide whose analysis wins when building silver data.
- **I5** (learn): every new component could be compared against a trivial bag-of-words control. This serves as a negative control to confirm that structural reasoning, not lexical overlap, explains the score.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 402 works · 212,763 citations
- [University of Rochester](https://md.research.elamur.ai/free/en/universities/university-of-rochester/) — University, Rochester, United States · 43 works · 7,874 citations

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,954 works · [🤝 Sociology and Political Science](https://md.research.elamur.ai/free/en/shelves/social-sciences/sociology-and-political-science/) · 626 works · [🤝 Communication](https://md.research.elamur.ai/free/en/shelves/social-sciences/communication/) · 104 works

🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,379 works

## Cite

Cite as: Thai Hoang and Phuong Vu. 2020. Not-NUTs at WNUT-2020 Task 2: A BERT-based System in Identifying Informative COVID-19 English Tweets. Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{hoang2020not,
  title = {{Not-NUTs at WNUT-2020 Task 2: A BERT-based System in Identifying Informative COVID-19 English Tweets}},
  author = {Thai Hoang and Phuong Vu},
  booktitle = {Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020)},
  year = {2020},
  doi = {10.18653/v1/2020.wnut-1.69},
  url = {https://doi.org/10.18653/v1/2020.wnut-1.69}
}
```

- [← previous Not-NUTs at W-NUT 2020 Task 2: A BERT-based System in Identifying Informative COVID-19 English Tweets](https://md.research.elamur.ai/free/en/papers/hoang-2020-nuts-w-nut-2020-task/)
- [next → Numeral differential constructions in Arabic](https://md.research.elamur.ai/free/en/papers/hallman-2020-numeral-differential-constructions-arabic/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/11/)

## References

12 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. [Unsupervised cross-lingual representation learning at scale](https://md.research.elamur.ai/free/en/papers/conneau-2020-unsupervised-cross-lingual-representation-learning/).
2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/).
3. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/).
4. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2019%2E+Decoupled+weight+decay+regularization%2E+In+International+Conference+on+Learning+Representations%2E)
5. Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020a. [BERTweet: A pre-trained language model for English Tweets](https://md.research.elamur.ai/free/en/papers/nguyen-2020-bertweet-pre-trained-language-model/). arXiv preprint, arXiv:2005.10200. [↗](https://arxiv.org/abs/2005.10200)
6. Dat Quoc Nguyen, Thanh Vu, Afshin Rahimi, Mai Hoang Dao, Linh The Nguyen, and Long Doan. 2020b. [WNUT-2020 Task 2: Identification of Informative COVID-19 English Tweets](https://md.research.elamur.ai/free/en/papers/nguyen-2020-wnut-2020-task-2-identification/). In Proceedings of the 6th Workshop on Noisy User-generated Text.
7. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. [find](https://scholar.google.com/scholar?q=Adam+Paszke%2C+Sam+Gross%2C+Soumith+Chintala%2C+Gregory+Chanan%2C+Edward+Yang%2C+Zachary+DeVito%2C+Zeming+Lin%2C+Alban+Desmaison%2C+Luca+Antiga%2C+and+Adam+Lerer%2E+2017%2E+Automatic+differentiation+in+pytorch%2E)
8. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. [find](https://scholar.google.com/scholar?q=F%2E+Pedregosa%2C+G%2E+Varoquaux%2C+A%2E+Gramfort%2C+V%2E+Michel%2C+B%2E+Thirion%2C+O%2E+Grisel%2C+M%2E+Blondel%2C+P%2E+Prettenhofer%2C+R%2E+Weiss%2C+V%2E+Dubourg%2C+J%2E+Vanderplas%2C+A%2E+Passos%2C+D%2E+Cournapeau%2C+M%2E+Brucher%2C+M%2E+Perrot%2C+and+E%2E+Duc)
9. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. [Deep contextualized word representations](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/).
10. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/).
11. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2019. Huggingface’s [transformers: State-of-the-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/).
12. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. [Xlnet: Generalized autoregressive pretraining for language understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/).

*Updated 2 November 2026*