---
title: "TextFlint — paper by Wang et al. (2021) — summary and ideas"
description: "Earlier robustness tools usually test either generic perturbations or task-specific ones, which makes a… 30 references, cited by 67 papers here."
authors: ["Xiao Wang", "Qin Liu", "Tao Gui", "Qi Zhang", "Yicheng Zou", "Xin Zhou", "Jiacheng Ye", "Yongxin Zhang", "Rui Zheng", "Zexiong Pang", "Qinzhuo Wu", "Zhengyan Li", "Chong Zhang", "Ruotian Ma", "Zichu Fei", "Ruijian Cai", "Jun Zhao", "Xingwu Hu", "Zhiheng Yan", "Yiding Tan", "Yuan Hu", "Qiyuan Bian", "Zhihua Liu", "Shan Qin", "Bolin Zhu", "Xiaoyu Xing", "Jinlan Fu", "Yue Zhang", "Minlong Peng", "Xiaoqing Zheng", "Yaqian Zhou", "Zhongyu Wei", "Xipeng Qiu", "Xuanjing Huang"]
year: 2021
venue: "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations"
source: "https://aclanthology.org/2021.acl-demo.41/"
doi: "10.18653/v1/2021.acl-demo.41"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2021-textflint-unified-multilingual-robustness-evaluation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2021-textflint-unified-multilingual-robustness-evaluation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) / TextFlint: Unified Multilingual Robustness Evaluation Toolkit for Natural Language Processing

# TextFlint: Unified Multilingual Robustness Evaluation Toolkit for Natural Language Processing

**Authors:** [Xiao Wang](https://md.research.elamur.ai/free/en/authors/xiao-wang/), [Qin Liu](https://md.research.elamur.ai/free/en/authors/qin-liu/), [Tao Gui](https://md.research.elamur.ai/free/en/authors/tao-gui/), [Qi Zhang](https://md.research.elamur.ai/free/en/authors/qi-zhang/), Yicheng Zou, [Xin Zhou](https://md.research.elamur.ai/free/en/authors/xin-zhou/), [Jiacheng Ye](https://md.research.elamur.ai/free/en/authors/jiacheng-ye/), Yongxin Zhang, [Rui Zheng](https://md.research.elamur.ai/free/en/authors/rui-zheng/), Zexiong Pang, Qinzhuo Wu, Zhengyan Li, [Chong Zhang](https://md.research.elamur.ai/free/en/authors/chong-zhang/), [Ruotian Ma](https://md.research.elamur.ai/free/en/authors/ruotian-ma/), Zichu Fei, Ruijian Cai, [Jun Zhao](https://md.research.elamur.ai/free/en/authors/jun-zhao/), Xingwu Hu, Zhiheng Yan, Yiding Tan, Yuan Hu, Qiyuan Bian, Zhihua Liu, Shan Qin, Bolin Zhu, Xiaoyu Xing, [Jinlan Fu](https://md.research.elamur.ai/free/en/authors/jinlan-fu/), [Yue Zhang](https://md.research.elamur.ai/free/en/authors/yue-zhang/), Minlong Peng, [Xiaoqing Zheng](https://md.research.elamur.ai/free/en/authors/xiaoqing-zheng/), [Yaqian Zhou](https://md.research.elamur.ai/free/en/authors/yaqian-zhou/), [Zhongyu Wei](https://md.research.elamur.ai/free/en/authors/zhongyu-wei/), [Xipeng Qiu](https://md.research.elamur.ai/free/en/authors/xipeng-qiu/) and [Xuanjing Huang](https://md.research.elamur.ai/free/en/authors/xuanjing-huang/) · **Year:** 2021 · **Venue:** [Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2021](https://md.research.elamur.ai/free/en/venues/acl/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.acl-demo.41/) · [PDF](https://aclanthology.org/2021.acl-demo.41.pdf) · [DOI 10.18653/v1/2021.acl-demo.41](https://doi.org/10.18653/v1/2021.acl-demo.41) · [OpenAlex](https://openalex.org/W3174150157)

**Citations:** 79 citations (OpenAlex, 2026-10-14) · [67](https://md.research.elamur.ai/free/en/papers/wang-2021-textflint-unified-multilingual-robustness-evaluation/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
NLP models with high accuracy often break when their inputs are slightly altered. Earlier robustness tools usually test either generic perturbations or task-specific ones, which makes a complete assessment hard. TextFlint addresses this with one open-source framework for English and Chinese. It combines four kinds of probes: universal text transformations, transformations tailored to particular tasks, adversarial attacks, and subpopulation slicing.

The system is a three-stage pipeline:
- an input layer that loads a JSON dataset, a config file, and a wrapped model (FlintModel);
- a generation layer with Transformation, Subpopulation, AttackRecipe, and a quality-checking Validator;
- a report layer that groups results by linguistic aspects.

The transformations are designed on linguistic grounds, and human annotators judged their outputs for plausibility and grammaticality. The authors ran more than 67,000 evaluations on 95 models and systems. Almost all of them degraded noticeably, and BERT sometimes lost over half its accuracy. The toolkit also produces augmentation data aimed at the weaknesses it finds. Overall, the work argues that robustness testing should become a standard part of NLP evaluation.

## Key points
- The toolkit offers 20 universal and 60 task-specific transformations, plus thousands of combinations, across 12 NLP tasks in English and Chinese.
- It ships 16 adversarial attack recipes built on TextAttack and four subpopulation options: GenderBias, TextLength, LanguageModelPerplexity, and PhraseMatching. For example, TextLength selects the longest or shortest 20% of samples.
- The Validator scores generated text with five metrics: GPT2 perplexity, word replacement ratio, edit distance, Universal Sentence Encoder similarity, and BLEU.
- The large-scale study covered 95 models on 6,903 transformed datasets, over 67,000 evaluations in total. BERT's accuracy fell by more than 50% on ABSA, NER, and NLI. On ABSA, BERT-Aspect dropped from 90.32 to 81.58 under the AddDiff transformation.
- Up to 100,000 transformed samples were rated by humans, and results are published at textflint.io.

## Interesting ideas
- **Mix generic and task-aware perturbations.** Typos and keyboard noise test surface stability, while task-specific edits probe task logic. An example of the latter is flipping the sentiment of a non-target aspect in ABSA. A robustness suite for any language could pair the two kinds in the same way.
- **Organize perturbations by linguistic level.** Grouping transformations under morphology, syntax, paradigmatic relations, and pragmatics makes the resulting report easier to interpret. This scheme could transfer well to morphologically rich languages, where errors on inflection or derivation deserve their own category.
- **Keep model-free generation separate from model-querying attacks.** Transformations and subpopulations never call the model, so they can be generated once on a CPU and reused across many models. Only attacks need model scores. This design keeps benchmarks cheap and reproducible.
- **Treat subpopulations as a diagnostic.** Slicing the existing test set by length, perplexity, gender cues, or phrase matches can reveal weak spots without creating any new text. It is a low-effort first check for any dataset.
- **Filter generated samples automatically, then confirm with people.** Scoring outputs on fluency, similarity, and edit size before human checking helps keep augmented or perturbed data trustworthy. The same approach could serve any synthetic-data pipeline.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Fudan University](https://md.research.elamur.ai/free/en/universities/fudan-university/) — University, Shanghai, China · 65 works · 19,402 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 467 works

## Cite

Cite as: Xiao Wang, Qin Liu, Tao Gui, Qi Zhang, Yicheng Zou, Xin Zhou, Jiacheng Ye, Yongxin Zhang, Rui Zheng, Zexiong Pang, Qinzhuo Wu, Zhengyan Li, Chong Zhang, Ruotian Ma, Zichu Fei, Ruijian Cai, Jun Zhao, Xingwu Hu, Zhiheng Yan, Yiding Tan, Yuan Hu, Qiyuan Bian, Zhihua Liu, Shan Qin, Bolin Zhu, Xiaoyu Xing, Jinlan Fu, Yue Zhang, Minlong Peng, Xiaoqing Zheng, Yaqian Zhou, Zhongyu Wei, Xipeng Qiu and Xuanjing Huang. 2021. TextFlint: Unified Multilingual Robustness Evaluation Toolkit for Natural Language Processing. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wang2021textflint,
  title = {{TextFlint: Unified Multilingual Robustness Evaluation Toolkit for Natural Language Processing}},
  author = {Xiao Wang, Qin Liu, Tao Gui, Qi Zhang, Yicheng Zou, Xin Zhou, Jiacheng Ye, Yongxin Zhang, Rui Zheng, Zexiong Pang, Qinzhuo Wu, Zhengyan Li, Chong Zhang, Ruotian Ma, Zichu Fei, Ruijian Cai, Jun Zhao, Xingwu Hu, Zhiheng Yan, Yiding Tan, Yuan Hu, Qiyuan Bian, Zhihua Liu, Shan Qin, Bolin Zhu, Xiaoyu Xing, Jinlan Fu, Yue Zhang, Minlong Peng, Xiaoqing Zheng, Yaqian Zhou, Zhongyu Wei, Xipeng Qiu and Xuanjing Huang},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations},
  year = {2021},
  doi = {10.18653/v1/2021.acl-demo.41},
  url = {https://doi.org/10.18653/v1/2021.acl-demo.41}
}
```

- [← previous Introducing CAD: the Contextual Abuse Dataset](https://md.research.elamur.ai/free/en/papers/vidgen-2021-introducing-cad-contextual-abuse-dataset/)
- [next → Word-Level Coreference Resolution](https://md.research.elamur.ai/free/en/papers/dobrovolskii-2021-word-level-coreference-resolution/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/)

## References

30 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dustin Arendt, Zhuanyi Huang, Prasha Shrestha, Ellyn Ayton, Maria Glenski, and Svitlana Volkova. 2020. Crosscheck: Rapid, reproducible, and interpretable model evaluation. [find](https://scholar.google.com/scholar?q=Dustin+Arendt%2C+Zhuanyi+Huang%2C+Prasha+Shrestha%2C+Ellyn+Ayton%2C+Maria+Glenski%2C+and+Svitlana+Volkova%2E+2020%2E+Crosscheck%3A+Rapid%2C+reproducible%2C+and+interpretable+model+evaluation%2E)
2. Vijay Arya, Rachel KE Bellamy, Pin-Yu Chen, Amit Dhurandhar, Michael Hind, Samuel C Hoffman, Stephanie Houde, Q Vera Liao, Ronny Luss, Aleksandra Mojsilović, et al. 2019. One explanation does not fit all: A toolkit and taxonomy of ai explainability techniques. arXiv preprint arXiv:1909.03012. — [arXiv:1909.03012](https://arxiv.org/abs/1909.03012) [find](https://scholar.google.com/scholar?q=Vijay+Arya%2C+Rachel+KE+Bellamy%2C+Pin%2DYu+Chen%2C+Amit+Dhurandhar%2C+Michael+Hind%2C+Samuel+C+Hoffman%2C+Stephanie+Houde%2C+Q+Vera+Liao%2C+Ronny+Luss%2C+Aleksandra+Mojsilovic%CC%81%2C+et+al%2E+2019%2E+One+explanation+does+not+fit)
3. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. 2018. [Universal sentence encoder for English](https://md.research.elamur.ai/free/en/papers/cer-2018-universal-sentence-encoder-english/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Langua…
4. Dehong Ma, Sujian Li, Xiaodong Zhang, and Houfeng Wang. 2017. Interactive attention networks for aspect-level sentiment classification. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pages 4068–4074. [find](https://scholar.google.com/scholar?q=Dehong+Ma%2C+Sujian+Li%2C+Xiaodong+Zhang%2C+and+Houfeng+Wang%2E+2017%2E+Interactive+attention+networks+for+aspect%2Dlevel+sentiment+classification%2E+In+Proceedings+of+the+26th+International+Joint+Conference+on+Art)
5. Edward Ma. 2019. Nlp augmentation. [https://github.com/makcedward/nlpaug](https://github.com/makcedward/nlpaug). [↗](https://github.com/makcedward/nlpaug)
6. John Miller, Karl Krauth, Benjamin Recht, and Ludwig Schmidt. 2020. The effect of natural distribution shift on question answering models. In International Conference on Machine Learning, pages 6905–6916. PMLR. [find](https://scholar.google.com/scholar?q=John+Miller%2C+Karl+Krauth%2C+Benjamin+Recht%2C+and+Ludwig+Schmidt%2E+2020%2E+The+effect+of+natural+distribution+shift+on+question+answering+models%2E+In+International+Conference+on+Machine+Learning%2C+pages+6905%E2%80%936)
7. John X Morris, Eli Lifland, Jin Yong Yoo, and Yanjun Qi. 2020. [Textattack: A framework for adversarial attacks in natural language processing](https://md.research.elamur.ai/free/en/papers/morris-2020-textattack-framework-adversarial-attacks-data/). arXiv preprint arXiv:2005.05909.
8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
9. Harsha Nori, Samuel Jenkins, Paul Koch, and Rich Caruana. 2019. Interpretml: A unified framework for machine learning interpretability. [find](https://scholar.google.com/scholar?q=Harsha+Nori%2C+Samuel+Jenkins%2C+Paul+Koch%2C+and+Rich+Caruana%2E+2019%2E+Interpretml%3A+A+unified+framework+for+machine+learning+interpretability%2E)
10. Feifan Fan, Yansong Feng, and Dongyan Zhao. 2018. [Multi-grained attention network for aspect-level sentiment classification](https://md.research.elamur.ai/free/en/papers/fan-2018-multi-grained-attention-network-aspect/). In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 3433–3442.
11. Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. 2020. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673. [find](https://scholar.google.com/scholar?q=Robert+Geirhos%2C+Jo%CC%88rn%2DHenrik+Jacobsen%2C+Claudio+Michaelis%2C+Richard+Zemel%2C+Wieland+Brendel%2C+Matthias+Bethge%2C+and+Felix+A+Wichmann%2E+2020%2E+Shortcut+learning+in+deep+neural+networks%2E+Nature+Machine+Intelli)
12. Karan Goel, Nazneen Rajani, Jesse Vig, Samson Tan, Jason Wu, Stephan Zheng, Caiming Xiong, Mohit Bansal, and Christopher Ré. 2021. Robustness gym: Unifying the nlp evaluation landscape. [find](https://scholar.google.com/scholar?q=Karan+Goel%2C+Nazneen+Rajani%2C+Jesse+Vig%2C+Samson+Tan%2C+Jason+Wu%2C+Stephan+Zheng%2C+Caiming+Xiong%2C+Mohit+Bansal%2C+and+Christopher+Re%CC%81%2E+2021%2E+Robustness+gym%3A+Unifying+the+nlp+evaluation+landscape%2E)
13. Sepp Hochreiter, Jürgen Schmidhuber, et al. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter%2C+Ju%CC%88rgen+Schmidhuber%2C+et+al%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A1735%E2%80%931780%2E)
14. Yoon Kim. 2014. [Convolutional neural networks for sentence classification](https://md.research.elamur.ai/free/en/papers/kim-2014-convolutional-neural-networks-sentence-classification/). In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746–1751, Doha, Qatar. Association for Computational Linguistics.
15. Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. 2020. [Bert-attack: Adversarial attack against bert using bert](https://md.research.elamur.ai/free/en/papers/li-2020-bert-attack-adversarial-attack-against/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6193–6202.
16. Xin Li, Lidong Bing, Wai Lam, and Bei Shi. 2018. [Transformation networks for target-oriented sentiment classification](https://md.research.elamur.ai/free/en/papers/li-2018-transformation-networks-target-oriented-sentiment/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 946–956.
17. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
18. Danish Pruthi, Bhuwan Dhingra, and Zachary C. Lipton. 2019. [Combating adversarial misspellings with robust word recognition](https://md.research.elamur.ai/free/en/papers/pruthi-2019-combating-adversarial-misspellings-robust-word/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5582–5591, Florence, Italy. Association for Computational Linguistics.
19. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
20. Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902–4912, Online. Association for Computational Linguistic… [find](https://scholar.google.com/scholar?q=Marco+Tulio+Ribeiro%2C+Tongshuang+Wu%2C+Carlos+Guestrin%2C+and+Sameer+Singh%2E+2020%2E+Beyond+accuracy%3A+Behavioral+testing+of+NLP+models+with+CheckList%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Associat)
21. Duyu Tang, Bing Qin, Xiaocheng Feng, and Ting Liu. 2016a. [Effective lstms for target-dependent sentiment classification](https://md.research.elamur.ai/free/en/papers/tang-2016-effective-lstms-target-dependent-sentiment/). In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3298– 3307.
22. Duyu Tang, Bing Qin, and Ting Liu. 2016b. [Aspect level sentiment classification with deep memory network](https://md.research.elamur.ai/free/en/papers/tang-2016-aspect-level-sentiment-classification-deep/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 214–224.
23. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. [Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition](https://md.research.elamur.ai/free/en/papers/sang-2003-introduction-conll-2003-shared-task/). In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142–147.
24. Eric Wallace, Jens Tuyls, Junlin Wang, Sanjay Subramanian, Matt Gardner, and Sameer Singh. 2019. [AllenNLP interpret: A framework for explaining predictions of NLP models](https://md.research.elamur.ai/free/en/papers/wallace-2019-allennlp-interpret-framework-explaining-predictions/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confere…
25. Yequan Wang, Minlie Huang, Xiaoyan Zhu, and Li Zhao. 2016. [Attention-based lstm for aspect-level sentiment classification](https://md.research.elamur.ai/free/en/papers/wang-2016-attention-based-lstm-aspect-level/). In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 606–615.
26. Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel S Weld. 2019. Errudite: Scalable, reproducible, and testable error analysis. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 747–763. [find](https://scholar.google.com/scholar?q=Tongshuang+Wu%2C+Marco+Tulio+Ribeiro%2C+Jeffrey+Heer%2C+and+Daniel+S+Weld%2E+2019%2E+Errudite%3A+Scalable%2C+reproducible%2C+and+testable+error+analysis%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+f)
27. Xiaoyu Xing, Zhijing Jin, Di Jin, Bingning Wang, Qi Zhang, and Xuan-Jing Huang. 2020. Tasty burgers, soggy fries: Probing aspect robustness in aspect-based sentiment analysis. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3594–3605. [find](https://scholar.google.com/scholar?q=Xiaoyu+Xing%2C+Zhijing+Jin%2C+Di+Jin%2C+Bingning+Wang%2C+Qi+Zhang%2C+and+Xuan%2DJing+Huang%2E+2020%2E+Tasty+burgers%2C+soggy+fries%3A+Probing+aspect+robustness+in+aspect%2Dbased+sentiment+analysis%2E+In+Proceedings+of+the+20)
28. Biqing Zeng, Heng Yang, Ruyang Xu, Wu Zhou, and Xuli Han. 2019. Lcf: A local context focus mechanism for aspect-based sentiment classification. Applied Sciences, 9(16):3389. [find](https://scholar.google.com/scholar?q=Biqing+Zeng%2C+Heng+Yang%2C+Ruyang+Xu%2C+Wu+Zhou%2C+and+Xuli+Han%2E+2019%2E+Lcf%3A+A+local+context+focus+mechanism+for+aspect%2Dbased+sentiment+classification%2E+Applied+Sciences%2C+9%2816%29%3A3389%2E)
29. Guoyang Zeng, Fanchao Qi, Qianrui Zhou, Tingji Zhang, Bairu Hou, Yuan Zang, Zhiyuan Liu, and Maosong Sun. 2020. Openattack: An open-source textual adversarial attack toolkit. arXiv preprint arXiv:2009.09191. [find](https://scholar.google.com/scholar?q=Guoyang+Zeng%2C+Fanchao+Qi%2C+Qianrui+Zhou%2C+Tingji+Zhang%2C+Bairu+Hou%2C+Yuan+Zang%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+2020%2E+Openattack%3A+An+open%2Dsource+textual+adversarial+attack+toolkit%2E+arXiv+preprint+arXiv%3A2009)
30. Jiawei Zhang, Yang Wang, Piero Molino, Lezhi Li, and David S Ebert. 2018. Manifold: A model-agnostic framework for interpretation and diagnosis of machine learning models. IEEE transactions on visualization and computer graphics, 25(1):364– 373. [find](https://scholar.google.com/scholar?q=Jiawei+Zhang%2C+Yang+Wang%2C+Piero+Molino%2C+Lezhi+Li%2C+and+David+S+Ebert%2E+2018%2E+Manifold%3A+A+model%2Dagnostic+framework+for+interpretation+and+diagnosis+of+machine+learning+models%2E+IEEE+transactions+on+visuali)

*Updated 24 October 2026*