---
title: "Comparing DAE-based and MASS-based UNMT: Robustness to"
description: "It compares MASS-style pretraining (span masking, no shuffling) with DAE-style pretraining (denoising with… 14.63 BLEU. 20 references."
authors: ["Tamali Banerjee", "Rudra Murthy", "Pushpak Bhattacharyya"]
year: 2023
venue: "Proceedings of the 20th International Conference on Natural Language Processing (ICON)"
source: "https://aclanthology.org/2023.icon-1.44/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/banerjee-2023-comparing-dae-based-mass-based/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/banerjee-2023-comparing-dae-based-mass-based/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/17/) / Comparing DAE-based and MASS-based UNMT: Robustness to Word-Order Divergence in English–\\ensuremath>Indic Language Pairs

# Comparing DAE-based and MASS-based UNMT: Robustness to Word-Order Divergence in English–\\ensuremath>Indic Language Pairs

**Authors:** Tamali Banerjee, [Rudra Murthy](https://md.research.elamur.ai/free/en/authors/rudra-murthy/) and [Pushpak Bhattacharyya](https://md.research.elamur.ai/free/en/authors/pushpak-bhattacharyya/) · **Year:** 2023 · **Venue:** [Proceedings of the 20th International Conference on Natural Language Processing (ICON)](https://md.research.elamur.ai/free/en/venues/icon/) ([ICON 2023](https://md.research.elamur.ai/free/en/venues/icon/2023/)) · **Publisher:** [NLP Association of India (NLPAI)](https://md.research.elamur.ai/free/en/publishers/nlp-association-of-india-nlpai/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.icon-1.44/) · [PDF](https://aclanthology.org/2023.icon-1.44.pdf)

## Summary
The paper asks whether two leading unsupervised NMT recipes cope with languages that order words differently. It compares MASS-style pretraining (span masking, no shuffling) with DAE-style pretraining (denoising with shuffle, mask and deletion noise) on English→Hindi, Bengali, Gujarati, Kannada and Tamil, which pair SVO English with SOV targets. Both use the static cross-lingual embedding variants of Banerjee et al. (2021).

Each approach is trained twice: once on original IndicCorp monolingual data, and once with English pre-reordered into SOV-like order using CFILT-pre-order rules. The logic is that a model robust to word order should gain little from reordering.

Reordering consistently helped MASS-static, which shows that MASS-static is sensitive to word order. DAE-static gained little or even lost quality, and DAE-static on original data was usually the best system overall. The authors trace the losses to lost case information and support the findings with an analysis of position embeddings. For low-resource, divergent pairs, the work suggests that shuffle noise can replace a brittle, one-directional reordering pipeline.

## Key points
- **Data:**
  - Monolingual training data comes from IndicCorp (e.g., 52.2M English and 63.1M Hindi sentences).
  - Validation and test sets come from WAT 2021: 711 and 1781 sentences per pair.
  - Sentences whose parse failed during reordering were removed from every condition.
- **Setup:**
  - Tokenization: Moses for English and the Indic NLP Library for the Indic languages.
  - Joint FastBPE with 100k merges; MASS code-base with word-mass 0.5.
  - DAE noise: word-shuffle 3, word-dropout 0.1, word-blank 0.1.
  - Metrics: BLEU and CHRF via SacreBLEU.
- **en→hi results:**
  - MASS-static rose from 14.16 to 14.63 BLEU with reordering.
  - DAE-static fell from 21.03 to 15.22 BLEU; CHRF dropped from 45.63 to 42.18.
- **Across pairs:** MASS-static improved on every pair after reordering, by up to about 6 CHRF. DAE-static changes were mixed and small, ranging from slight gains to clear drops.
- **Limitations and analysis:**
  - Reordered models only work in the source→target direction, because their reverse output would be reordered English.
  - Position-embedding cosine similarity is local (2–3 positions) in MASS-static and spans a wider neighbourhood in DAE-static.

## Interesting ideas
- Test robustness by training the same model on original input and on input transformed to remove a suspected difficulty; a small gap means the model already handles that difficulty.
- Before comparing two conditions, drop from both arms any sentence where the preprocessing in one arm failed, so that score differences are not caused by different data.
- Reordering English to SOV lost the case cue: the subject 'we' should have been rendered in the Hindi dative 'hameM', but the reordered model produced the nominative 'hama'. This shows that word position alone cannot carry role or case information.
- Training with local shuffle noise made the model treat nearby positions as similar, which gave flexibility toward order differences.
- Plotting cosine similarity between learned position embeddings shows how far a model's sense of position is blurred.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Run paired ablations, for example on parser input with normalized versus original word order, as a negative control that shows whether a component truly depends on order cues.
- **I2** (learn): When silver data from witness parsers is compared across pipelines, evaluate only on the intersection of items that every pipeline processed successfully.
- **I3** (back): When generating or translating from the meaning graph, choose target case from semantic role and verb frame rather than from linearized position.
- **I4** (tree): Augment gparse training with small local token shuffles to test or improve robustness on noisy or free-order input, while keeping gold arcs attached to the moved tokens.
- **I5** (tree): Use the same plot as an explainable diagnostic of gparse's positional encoding when comparing training regimes.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works

🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,379 works

## Cite

Cite as: Tamali Banerjee, Rudra Murthy and Pushpak Bhattacharyya. 2023. Comparing DAE-based and MASS-based UNMT: Robustness to Word-Order Divergence in English–\\ensuremath>Indic Language Pairs. Proceedings of the 20th International Conference on Natural Language Processing (ICON).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{banerjee2023comparing,
  title = {{Comparing DAE-based and MASS-based UNMT: Robustness to Word-Order Divergence in English–ensuremath>Indic Language Pairs}},
  author = {Tamali Banerjee, Rudra Murthy and Pushpak Bhattacharyya},
  booktitle = {Proceedings of the 20th International Conference on Natural Language Processing (ICON)},
  year = {2023},
  url = {https://aclanthology.org/2023.icon-1.44/}
}
```

- [← previous Chinese Semantic Model and Semantic Extraction Method Based on Dependency Syntax](https://md.research.elamur.ai/free/en/papers/wang-2023-chinese-semantic-model/)
- [next → Computing the optimal keyboard through a geometric analysis of the English language](https://md.research.elamur.ai/free/en/papers/deschamps-2023-computing-optimal-keyboard-through-geometric/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/17/)

## References

20 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018. [Unsupervised statistical machine translation](https://md.research.elamur.ai/free/en/papers/artetxe-2018-unsupervised-statistical-machine-translation/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3632– 3642.
2. Tamali Banerjee, Rudra V Murthy, and Pushpak Bhattacharya. 2021. Crosslingual embeddings are essential in UNMT for distant languages: An English to IndoAryan case study. In Proceedings of Machine Translation Summit XVIII: Research Track, pages 23– 34, Virtual. Association for Machine Translation in the Americas. [find](https://scholar.google.com/scholar?q=Tamali+Banerjee%2C+Rudra+V+Murthy%2C+and+Pushpak+Bhattacharya%2E+2021%2E+Crosslingual+embeddings+are+essential+in+UNMT+for+distant+languages%3A+An+English+to+IndoAryan+case+study%2E+In+Proceedings+of+Machine+Tran)
3. Pushpak Bhattacharyya. 2012. Natural language processing: A perspective from computation in presence of ambiguity, resource constraint and multilinguality. CSI journal of computing, 1(2):1–13. [find](https://scholar.google.com/scholar?q=Pushpak+Bhattacharyya%2E+2012%2E+Natural+language+processing%3A+A+perspective+from+computation+in+presence+of+ambiguity%2C+resource+constraint+and+multilinguality%2E+CSI+journal+of+computing%2C+1%282%29%3A1%E2%80%9313%2E)
4. Rajen Chatterjee, Anoop Kunchukuttan, and Pushpak Bhattacharyya. 2014. Supertag based pre-ordering in machine translation. In Proceedings of the 11th International Conference on Natural Language Processing, pages 30–38. [find](https://scholar.google.com/scholar?q=Rajen+Chatterjee%2C+Anoop+Kunchukuttan%2C+and+Pushpak+Bhattacharyya%2E+2014%2E+Supertag+based+pre%2Dordering+in+machine+translation%2E+In+Proceedings+of+the+11th+International+Conference+on+Natural+Language+Proce)
5. Alexandra Chronopoulou, Dario Stojanovski, and Alexander Fraser. 2021. Improving the lexical ability of pretrained language models for unsupervised neural machine translation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 173–180, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Alexandra+Chronopoulou%2C+Dario+Stojanovski%2C+and+Alexander+Fraser%2E+2021%2E+Improving+the+lexical+ability+of+pretrained+language+models+for+unsupervised+neural+machine+translation%2E+In+Proceedings+of+the+20)
6. Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In Advances in Neural Information Processing Systems, pages 7057– 7067. [find](https://scholar.google.com/scholar?q=Alexis+Conneau+and+Guillaume+Lample%2E+2019%2E+Cross%2Dlingual+language+model+pretraining%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+7057%E2%80%93+7067%2E)
7. Jinhua Du and Andy Way. 2017. Pre-reordering for neural machine translation: Helpful or harmful? The Prague bulletin of mathematical linguistics, 108(1):171. [find](https://scholar.google.com/scholar?q=Jinhua+Du+and+Andy+Way%2E+2017%2E+Pre%2Dreordering+for+neural+machine+translation%3A+Helpful+or+harmful%3F+The+Prague+bulletin+of+mathematical+linguistics%2C+108%281%29%3A171%2E)
8. Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Khapra, and Pratyush Kumar. 2020. [IndicNLPSuite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for Indian languages](https://md.research.elamur.ai/free/en/papers/kakwani-2020-indicnlpsuite-monolingual-corpora-evaluation-benchmarks/). In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4948– 4961, Online. Association for Computational Linguistics. toolkit for statistical machine translation. In Proceedings of the 45th annual meeting of the ACL on interactive poster and demonstration sessions, pages 177–180. Association for Computational Linguistics.
9. Anoop Kunchukuttan. 2020. The IndicNLP Library. [https://github.com/anoopkunchukuttan/](https://github.com/anoopkunchukuttan/) indic_nlp_library/blob/master/docs/ indicnlp.pdf. [↗](https://github.com/anoopkunchukuttan/)
10. Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018. Phrase-based & neural unsupervised machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5039–5049, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Guillaume+Lample%2C+Myle+Ott%2C+Alexis+Conneau%2C+Ludovic+Denoyer%2C+and+Marc%E2%80%99Aurelio+Ranzato%2E+2018%2E+Phrase%2Dbased+%26+neural+unsupervised+machine+translation%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+)
11. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742. [↗](https://doi.org/10.1162/tacl_a_00343)
12. Rudra Murthy V, Anoop Kunchukuttan, Pushpak Bhattacharyya, et al. 2019. Addressing word-order divergence in multilingual neural machine translation for extremely low resource languages. In Annual Conference of the North American Chapter of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rudra+Murthy+V%2C+Anoop+Kunchukuttan%2C+Pushpak+Bhattacharyya%2C+et+al%2E+2019%2E+Addressing+word%2Dorder+divergence+in+multilingual+neural+machine+translation+for+extremely+low+resource+languages%2E+In+Annual+Conf)
13. Toshiaki Nakazawa, Hideki Nakayama, Chenchen Ding, Raj Dabre, Shohei Higashiyama, Hideya Mino, Isao Goto, Win Pa Pa, Anoop Kunchukuttan, Shantipriya Parida, Ondřej Bojar, Chenhui Chu, Akiko Eriguchi, Kaori Abe, Yusuke Oda, and Sadao Kurohashi. 2021. Overview of the 8th workshop on Asian translation. In Proceedings of the 8th Workshop on Asian Translation (WAT2021), pages 1–45, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Toshiaki+Nakazawa%2C+Hideki+Nakayama%2C+Chenchen+Ding%2C+Raj+Dabre%2C+Shohei+Higashiyama%2C+Hideya+Mino%2C+Isao+Goto%2C+Win+Pa+Pa%2C+Anoop+Kunchukuttan%2C+Shantipriya+Parida%2C+Ondr%CC%8Cej+Bojar%2C+Chenhui+Chu%2C+Akiko+Eriguchi%2C)
14. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics. [↗](https://doi.org/10.3115/1073083.1073135)
15. Maja Popović. 2016. [chrf deconstructed: beta parameters and n-gram weights](https://md.research.elamur.ai/free/en/papers/popovic-2016-chrf-deconstructed-beta-parameters-n/). In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 499–504.
16. Jyotsana Khatri, Rudra Murthy, Tamali Banerjee, and Pushpak Bhattacharyya. 2021. Simple measures of bridging lexical divergence help unsupervised neural machine translation for low-resource languages. Machine Translation, 35(4):711–744. [find](https://scholar.google.com/scholar?q=Jyotsana+Khatri%2C+Rudra+Murthy%2C+Tamali+Banerjee%2C+and+Pushpak+Bhattacharyya%2E+2021%2E+Simple+measures+of+bridging+lexical+divergence+help+unsupervised+neural+machine+translation+for+low%2Dresource+languages%2E)
17. Matt Post. 2018. [A call for clarity in reporting BLEU scores](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/). In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Brussels, Belgium. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/w18-6319)
18. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019. [Mass: Masked sequence to sequence pre-training for language generation](https://md.research.elamur.ai/free/en/papers/song-2019-mass-masked-sequence-sequence-pre/). In International Conference on Machine Learning, pages 5926–5936.
19. Haipeng Sun, Rui Wang, Masao Utiyama, Benjamin Marie, Kehai Chen, Eiichiro Sumita, and Tiejun Zhao. 2021. Unsupervised neural machine translation for similar and distant language pairs: An empirical study. ACM Transactions on Asian and Low-Resource Language Information Processing (TAL-LIP), 20(1):1–17. [find](https://scholar.google.com/scholar?q=Haipeng+Sun%2C+Rui+Wang%2C+Masao+Utiyama%2C+Benjamin+Marie%2C+Kehai+Chen%2C+Eiichiro+Sumita%2C+and+Tiejun+Zhao%2E+2021%2E+Unsupervised+neural+machine+translation+for+similar+and+distant+language+pairs%3A+An+empirical+s)
20. Yang Zhao, Jiajun Zhang, and Chengqing Zong. 2018. Exploiting pre-ordering for neural machine translation. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Yang+Zhao%2C+Jiajun+Zhang%2C+and+Chengqing+Zong%2E+2018%2E+Exploiting+pre%2Dordering+for+neural+machine+translation%2E+In+Proceedings+of+the+Eleventh+International+Conference+on+Language+Resources+and+Evaluation+)

*Updated 2 November 2026*