---
title: "Spelling Error Correction with Soft-Masked BERT — Zhang 2020"
description: "Their proposed Soft-Masked BERT adds a Bi-GRU detection network. 17 references, cited by 68 papers here."
authors: ["Shaohua Zhang", "Haoran Huang", "Jicong Liu", "Hang Li"]
year: 2020
venue: "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/2020.acl-main.82/"
doi: "10.18653/v1/2020.acl-main.82"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2020-spelling-error-correction-soft-masked/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2020-spelling-error-correction-soft-masked/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/2/) / Spelling Error Correction with Soft-Masked BERT

# Spelling Error Correction with Soft-Masked BERT

**Authors:** Shaohua Zhang, Haoran Huang, Jicong Liu and [Hang Li](https://md.research.elamur.ai/free/en/authors/hang-li/) · **Year:** 2020 · **Venue:** [Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2020](https://md.research.elamur.ai/free/en/venues/acl/2020/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.acl-main.82/) · [PDF](https://aclanthology.org/2020.acl-main.82.pdf) · [DOI 10.18653/v1/2020.acl-main.82](https://doi.org/10.18653/v1/2020.acl-main.82) · [OpenAlex](https://openalex.org/W3034797320)

**Citations:** 215 citations (OpenAlex, 2026-10-14) · 68 in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper addresses Chinese spelling error correction (CSC) at the character level. In this task, a sentence must be mapped to a corrected sentence of the same length. The prior state of the art fine-tunes BERT to pick a character, or keep the original, at every position. The authors observe that this approach rarely flags errors. They attribute this to masked language model pre-training, where only about 15% of tokens are masked, so the model tends to copy its input.

Their proposed Soft-Masked BERT adds a Bi-GRU detection network. This network estimates an error probability for each character. Each input embedding is then blended with the [MASK] embedding in proportion to that probability before the result enters a BERT correction network. A residual connection links the input embeddings to BERT's final layer, and both networks are trained jointly with combined detection and correction losses.

Experiments use the SIGHAN benchmark and a newly built News Title dataset. Soft-Masked BERT clearly outperforms baselines, including plain BERT fine-tuning. The soft-masking idea could carry over to other detect-then-correct problems.

## Key points
- Detection is handled by a Bi-GRU with a sigmoid output. Soft masking computes e′ = p·e_mask + (1−p)·e, which becomes standard hard masking when p = 1.
- The correction network is a 12-block BERT with a softmax over all characters. Its training loss linearly combines the detection loss and the correction loss.
- Training pairs can be produced by injecting errors into clean text using a confusion table.
- The new News Title dataset consists of news headlines and is roughly ten times larger than earlier CSC datasets.
- In an analysis of 100 sampled errors, 67% required stronger reasoning and 11% required world knowledge. The remaining 22% had no clear type. An oracle "BERT-Finetune + Force" setup indicates substantial room for further gains.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Fudan University](https://md.research.elamur.ai/free/en/universities/fudan-university/) — University, Shanghai, China · 65 works · 19,402 citations
- [ByteDance](https://md.research.elamur.ai/free/en/companies/bytedance/) — Company, Beijing, China · 18 works · 3,890 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

## Cite

Cite as: Shaohua Zhang, Haoran Huang, Jicong Liu and Hang Li. 2020. Spelling Error Correction with Soft-Masked BERT. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhang2020spelling,
  title = {{Spelling Error Correction with Soft-Masked BERT}},
  author = {Shaohua Zhang, Haoran Huang, Jicong Liu and Hang Li},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.82},
  url = {https://doi.org/10.18653/v1/2020.acl-main.82}
}
```

- [← previous USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation](https://md.research.elamur.ai/free/en/papers/mehri-2020-usr-unsupervised-reference-free-evaluation/)
- [next → Fine-grained Fact Verification with Kernel Graph Attention Network](https://md.research.elamur.ai/free/en/papers/liu-2020-fine-grained-fact-verification-kernel/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/2/)

## References

17 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Haithem Afli, Zhengwei Qiu, Andy Way, and Páraic Sheridan. 2016. Using smt for ocr error correction of historical texts. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 962–966. [find](https://scholar.google.com/scholar?q=Haithem+Afli%2C+Zhengwei+Qiu%2C+Andy+Way%2C+and+Pa%CC%81raic+Sheridan%2E+2016%2E+Using+smt+for+ocr+error+correction+of+historical+texts%2E+In+Proceedings+of+the+Tenth+International+Conference+on+Language+Resources+and)
2. Jill Burstein and Martin Chodorow. 1999. Automated essay scoring for nonnative english speakers. In Proceedings of a Symposium on Computer Mediated Language Assessment and Evaluation in Natural Language Processing, pages 68–75. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jill+Burstein+and+Martin+Chodorow%2E+1999%2E+Automated+essay+scoring+for+nonnative+english+speakers%2E+In+Proceedings+of+a+Symposium+on+Computer+Mediated+Language+Assessment+and+Evaluation+in+Natural+Langua)
3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805.
4. Jianfeng Gao, Xiaolong Li, Daniel Micol, Chris Quirk, and Xu Sun. 2010. A large scale ranker-based system for search query spelling correction. In COL-ING 2010, 23rd International Conference on Computational Linguistics, Proceedings of the Conference, 23-27 August 2010, Beijing, China, pages 358– 36… [find](https://scholar.google.com/scholar?q=Jianfeng+Gao%2C+Xiaolong+Li%2C+Daniel+Micol%2C+Chris+Quirk%2C+and+Xu+Sun%2E+2010%2E+A+large+scale+ranker%2Dbased+system+for+search+query+spelling+correction%2E+In+COL%2DING+2010%2C+23rd+International+Conference+on+Comput)
5. Jinxi Guo, Tara N Sainath, and Ron J Weiss. 2019. A spelling correction model for end-to-end speech recognition. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5651–5655. IEEE. [find](https://scholar.google.com/scholar?q=Jinxi+Guo%2C+Tara+N+Sainath%2C+and+Ron+J+Weiss%2E+2019%2E+A+spelling+correction+model+for+end%2Dto%2Dend+speech+recognition%2E+In+ICASSP+2019%2D2019+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Proce)
6. Yuzhong Hong, Xianguo Yu, Neng He, Nan Liu, and Junhui Liu. 2019. Faspell: A fast, adaptable, simple, powerful chinese spell checker based on dae-decoder paradigm. In Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019), pages 160–169. [find](https://scholar.google.com/scholar?q=Yuzhong+Hong%2C+Xianguo+Yu%2C+Neng+He%2C+Nan+Liu%2C+and+Junhui+Liu%2E+2019%2E+Faspell%3A+A+fast%2C+adaptable%2C+simple%2C+powerful+chinese+spell+checker+based+on+dae%2Ddecoder+paradigm%2E+In+Proceedings+of+the+5th+Workshop+o)
7. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942. [find](https://scholar.google.com/scholar?q=Zhenzhong+Lan%2C+Mingda+Chen%2C+Sebastian+Goodman%2C+Kevin+Gimpel%2C+Piyush+Sharma%2C+and+Radu+Soricut%2E+2019%2E+Albert%3A+A+lite+bert+for+self%2Dsupervised+learning+of+language+representations%2E+arXiv+preprint+arXiv%3A1)
8. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+a)
9. Bruno Martins and Mário J. Silva. 2004. Spelling correction for search engine queries. In Advances in Natural Language Processing, 4th International Conference, EsTAL 2004, Alicante, Spain, October 20-22, 2004, Proceedings, pages 372–383. [find](https://scholar.google.com/scholar?q=Bruno+Martins+and+Ma%CC%81rio+J%2E+Silva%2E+2004%2E+Spelling+correction+for+search+engine+queries%2E+In+Advances+in+Natural+Language+Processing%2C+4th+International+Conference%2C+EsTAL+2004%2C+Alicante%2C+Spain%2C+October+2)
10. Yuen-Hsien Tseng, Lung-Hao Lee, Li-Ping Chang, and Hsin-Hsi Chen. 2015. Introduction to sighan 2015 bake-off for chinese spelling check. In Proceedings of the Eighth SIGHAN Workshop on Chinese Language Processing, pages 32–37. [find](https://scholar.google.com/scholar?q=Yuen%2DHsien+Tseng%2C+Lung%2DHao+Lee%2C+Li%2DPing+Chang%2C+and+Hsin%2DHsi+Chen%2E+2015%2E+Introduction+to+sighan+2015+bake%2Doff+for+chinese+spelling+check%2E+In+Proceedings+of+the+Eighth+SIGHAN+Workshop+on+Chinese+Languag)
11. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018a. [Glue: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). arXiv preprint arXiv:1804.07461.
12. Dingmin Wang, Yan Song, Jing Li, Jialong Han, and Haisong Zhang. 2018b. [A hybrid approach to automatic corpus generation for chinese spelling check](https://md.research.elamur.ai/free/en/papers/wang-2018-hybrid-approach-automatic-corpus-generation/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2517–2527.
13. Dingmin Wang, Yi Tay, and Li Zhong. 2019. Confusionset-guided pointer networks for chinese spelling check. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5780–5785. [find](https://scholar.google.com/scholar?q=Dingmin+Wang%2C+Yi+Tay%2C+and+Li+Zhong%2E+2019%2E+Confusionset%2Dguided+pointer+networks+for+chinese+spelling+check%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+p)
14. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. [Xlnet: Generalized autoregressive pretraining for language understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/). arXiv preprint arXiv:1906.08237.
15. Junjie Yu and Zhenghua Li. 2014. Chinese spelling error detection and correction based on language model, pronunciation, and shape. In Proceedings of The Third CIPS-SIGHAN Joint Conference on Chinese Language Processing, pages 220–223, Wuhan, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Junjie+Yu+and+Zhenghua+Li%2E+2014%2E+Chinese+spelling+error+detection+and+correction+based+on+language+model%2C+pronunciation%2C+and+shape%2E+In+Proceedings+of+The+Third+CIPS%2DSIGHAN+Joint+Conference+on+Chinese+)
16. Liang-Chih Yu, Lung-Hao Lee, Yuen-Hsien Tseng, and Hsin-Hsi Chen. 2014. Overview of sighan 2014 bake-off for chinese spelling check. In Proceedings of The Third CIPS-SIGHAN Joint Conference on Chinese Language Processing, pages 126–132. [find](https://scholar.google.com/scholar?q=Liang%2DChih+Yu%2C+Lung%2DHao+Lee%2C+Yuen%2DHsien+Tseng%2C+and+Hsin%2DHsi+Chen%2E+2014%2E+Overview+of+sighan+2014+bake%2Doff+for+chinese+spelling+check%2E+In+Proceedings+of+The+Third+CIPS%2DSIGHAN+Joint+Conference+on+Chinese)
17. Shuiyuan Zhang, Jinhua Xiong, Jianpeng Hou, Qiao Zhang, and Xueqi Cheng. 2015. Hanspeller++: A unified framework for chinese spelling correction. In Proceedings of the Eighth SIGHAN Workshop on Chinese Language Processing, pages 38–45. [find](https://scholar.google.com/scholar?q=Shuiyuan+Zhang%2C+Jinhua+Xiong%2C+Jianpeng+Hou%2C+Qiao+Zhang%2C+and+Xueqi+Cheng%2E+2015%2E+Hanspeller%2B%2B%3A+A+unified+framework+for+chinese+spelling+correction%2E+In+Proceedings+of+the+Eighth+SIGHAN+Workshop+on+Chines)

*Updated 24 October 2026*