---
title: "#GCDH at WNUT-2020 Task 2: BERT-Based Models for the"
description: "This system paper tackles WNUT-2020 Task 2, a binary classification task. 0.80 F1. 4 references, cited by 1 paper here."
authors: ["Hanna Varachkina", "Stefan Ziehe", "Tillmann Dönicke", "Franziska Pannach"]
year: 2020
venue: "Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020)"
source: "https://aclanthology.org/2020.wnut-1.68/"
doi: "10.18653/v1/2020.wnut-1.68"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/varachkina-2020-gcdh-wnut-2020-task-2/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/varachkina-2020-gcdh-wnut-2020-task-2/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/10/) / #GCDH at WNUT-2020 Task 2: BERT-Based Models for the Detection of Informativeness in English COVID-19 Related Tweets

# #GCDH at WNUT-2020 Task 2: BERT-Based Models for the Detection of Informativeness in English COVID-19 Related Tweets

**Authors:** [Hanna Varachkina](https://md.research.elamur.ai/free/en/authors/hanna-varachkina/), Stefan Ziehe, [Tillmann Dönicke](https://md.research.elamur.ai/free/en/authors/tillmann-donicke/) and [Franziska Pannach](https://md.research.elamur.ai/free/en/authors/franziska-pannach/) · **Year:** 2020 · **Venue:** [Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020)](https://md.research.elamur.ai/free/en/venues/wnut/) ([Workshop on Noisy User-generated Text 2020](https://md.research.elamur.ai/free/en/venues/wnut/2020/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.wnut-1.68/) · [PDF](https://aclanthology.org/2020.wnut-1.68.pdf) · [DOI 10.18653/v1/2020.wnut-1.68](https://doi.org/10.18653/v1/2020.wnut-1.68) · [OpenAlex](https://openalex.org/W3101547274)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/varachkina-2020-gcdh-wnut-2020-task-2/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
This system paper tackles WNUT-2020 Task 2, a binary classification task. Systems had to separate English COVID-19 tweets that report case statistics (confirmed, suspected, recovered, deaths) from tweets that carry no such data. The motivation is to feed real-time trackers, since official agencies publish figures only about once a day.

The team used the organizers' split of 7,000 training and 1,000 validation tweets. They first built two classic baselines over token n-grams: a Naive Bayes classifier and an SVM, plus many hand-made extra features. They then fine-tuned two transformers, uncased BERT-large and COVID-Twitter-BERT (CT-BERT). Both baselines plateaued near 0.80 F1. The transformers gained more than ten points, and the domain-adapted CT-BERT came out on top (0.9231 F1 on validation, 0.9091 on the official test).

The work shows that pretraining on in-domain social media text beats generic pretraining. It also shows that feature engineering added nothing over plain n-grams. The authors argue that informativeness detection is only a first step toward checking claims and their timeliness.

## Key points
- **Baselines:**
  - NBC peaked at 0.8012 F1 using uni- and bigrams without number masking.
  - SVM reached 0.8009 with binary uni-/bigram features and masked numbers.
  - None of the extra SVM features improved results: VADER sentiment, Gunning fog index, CORD-19 medical-term counts, Wang et al. (2015) content features, URL/USER position flags and an opinion-phrase list.
- **Transformer setup:**
  - Preprocessing used the CT-BERT code. User names and links became placeholder tokens, emoji became ASCII, and 64 tweets that ended up empty were dropped.
  - 24 tweets were shifted to validation so batches stayed exactly 32.
  - Training ran on Colab TPUs with sequence length 96, Adam at 2e-5, and per-layer learning-rate decay of 0.95.
- **Validation results:** BERT-large (2 epochs) scored 0.8928 F1. CT-BERT-1 (3 epochs) scored 0.9231, with recall 0.9487.
- **Official test results:** CT-BERT-1 (6,912 tweets) got 0.9091 F1. CT-BERT-2, also trained on validation data (7,936 tweets), got 0.9036: precision rose slightly but F1 did not. The roughly 2,000 test tweets were hidden among about 12,000.
- **Error analysis:** Most of BERT's 115 errors were uninformative tweets labelled informative (87). CT-BERT-1 cut that number to 52. The remaining errors mixed statistics with opinion, personal stories, politics or protective measures.

## Interesting ideas
- Replacing all numbers with one placeholder while sparing the domain-meaningful "19" in "COVID-19" lets n-gram models generalize over figures without destroying a name.
- Before adding a feature, the authors plotted its distribution and found the data was already almost entirely neutral in sentiment, so the feature could not discriminate.
- Giving earlier transformer layers smaller learning rates (decay factor 0.95 per layer) protects general pretrained knowledge while the top layers adapt to the task.
- Hiding the real test items inside a much larger pool stops participants from tuning on the test set.
- The authors note that a true, informative message may still be stale, such as yesterday's case count, so validity depends on time.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tokens): The en tokenizer could offer a number-abstraction view with a whitelist of numeric tokens that belong to names, keeping both the raw value and the class for later stages.
- **I2** (learn): Run a quick distribution check on each candidate feature or repair-rule trigger in silver data, and drop those that are near-constant before spending training effort.
- **I3** (tree): gparse fine-tuning on a new treebank or UD dialect could apply layer-wise decay to keep low-level syntactic representations stable.
- **I4** (learn): Evaluation of readers or parsers could embed gold items among unlabeled decoys, a cheap negative control against overfitting to known benchmarks.
- **I5** (world): Facts and quantities in the text world could carry a reference time, so answers can flag outdated states instead of treating every reported number as current.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Göttingen](https://md.research.elamur.ai/free/en/universities/university-of-gottingen/) — University, Göttingen, Germany · 20 works · 158 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,954 works · [🤝 Sociology and Political Science](https://md.research.elamur.ai/free/en/shelves/social-sciences/sociology-and-political-science/) · 626 works

🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,379 works

## Cite

Cite as: Hanna Varachkina, Stefan Ziehe, Tillmann Dönicke and Franziska Pannach. 2020. #GCDH at WNUT-2020 Task 2: BERT-Based Models for the Detection of Informativeness in English COVID-19 Related Tweets. Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{varachkina2020gcdh,
  title = {{\#GCDH at WNUT-2020 Task 2: BERT-Based Models for the Detection of Informativeness in English COVID-19 Related Tweets}},
  author = {Hanna Varachkina, Stefan Ziehe, Tillmann Dönicke and Franziska Pannach},
  booktitle = {Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020)},
  year = {2020},
  doi = {10.18653/v1/2020.wnut-1.68},
  url = {https://doi.org/10.18653/v1/2020.wnut-1.68}
}
```

- [← previous Utilizing Subword Entities in Character-Level Sequence-to-Sequence Lemmatization Models](https://md.research.elamur.ai/free/en/papers/zalmout-2020-utilizing-subword-entities-character-level/)
- [next → Adapting the ISO 24617-2 Dialogue Act Annotation Scheme for Modelling Medical Consultations](https://md.research.elamur.ai/free/en/papers/petukhova-2020-adapting-iso-24617-2-dialogue/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/10/)

## References

4 references, 3 of them with a card of ours. ↗ leads to the work.

1. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805. [↗](https://arxiv.org/abs/1810.04805)
2. Diederik P. Kingma and Jimmy Ba. 2015. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). arXiv preprint arXiv:1412.6980. [↗](https://arxiv.org/abs/1412.6980)
3. Martin Müller, Marcel Salathé, and Per E. Kummervold. 2020. COVID-Twitter-BERT: A natural language processing model to analyse COVID-19 content on Twitter. arXiv preprint arXiv:2005.07503. [↗](https://arxiv.org/abs/2005.07503)
4. Dat Quoc Nguyen, Thanh Vu, Afshin Rahimi, Mai Hoang Dao, Linh The Nguyen, and Long Doan. 2020. [WNUT-2020 Task 2: Identification of Informative COVID-19 English Tweets](https://md.research.elamur.ai/free/en/papers/nguyen-2020-wnut-2020-task-2-identification/). In Proceedings of the 6th Workshop on Noisy User-generated Text.

*Updated 31 October 2026*