---
title: "Can You Unpack That? Learning to Rewrite — Elgohary 2019"
description: "In conversational question answering, later questions depend on earlier turns."
authors: ["Ahmed Elgohary", "Denis Peskov", "Jordan Boyd-Graber"]
year: 2019
venue: "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)"
source: "https://aclanthology.org/D19-1605/"
doi: "10.18653/v1/d19-1605"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/elgohary-2019-can-you-unpack-that-learning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/elgohary-2019-can-you-unpack-that-learning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/) / Can You Unpack That? Learning to Rewrite Questions-in-Context

# Can You Unpack That? Learning to Rewrite Questions-in-Context

**Authors:** [Ahmed Elgohary](https://md.research.elamur.ai/free/en/authors/ahmed-elgohary/), Denis Peskov, [Jordan Boyd-Graber](https://md.research.elamur.ai/free/en/authors/jordan-boyd-graber/) · **Year:** 2019 · **Venue:** Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D19-1605/) · [PDF](https://aclanthology.org/D19-1605.pdf) · [DOI 10.18653/v1/d19-1605](https://doi.org/10.18653/v1/d19-1605) · [OpenAlex](https://openalex.org/W2970996870)

**Citations:** 150 citations (openalex-snapshot · 2026-10-14) · [94](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
In conversational question answering, later questions depend on earlier turns. Pronouns, dropped entities, and phrases like "anything else" only make sense given the dialogue history. The authors define a task called question-in-context rewriting: converting such a dependent question into a standalone version with the same answer. To support it, they built CANARD by crowdsourcing rewrites of QuAC questions on Mechanical Turk. Workers were told to make minimal edits, live JavaScript checks warned about leftover pronouns or vague words, and the authors manually reviewed flagged submissions. They compare three baselines using BLEU against human rewrites: an unchanged copy, a rule replacing the first pronoun with the article title, and an attention-based Seq2Seq LSTM with a copy mechanism. The neural model beats the simple baselines but stays roughly 9 BLEU below human agreement. Its typical errors are overly short rewrites, confusion between related entities, and dropped list items. The resource matters because it turns conversational QA into ordinary reading comprehension and supports future systems that ask users for clarification.

## Key points
- CANARD contains 40,527 rewritten questions derived from QuAC: 31,538 for training, 3,418 for development (from QuAC train), and 5,571 for testing (QuAC's full dev set).
- Quality control flagged about 15% of the data for review, and 19.9% of submissions were rejected. In a manual check of 50 rewrites, every desired property appeared in at least 90% of cases.
- BLEU scores (dev/test):
  - Copy: 33.84/36.25
  - Pronoun Sub: 47.72/47.44
  - Seq2Seq (OpenNMT, GloVE embeddings): 51.37/49.67
  - Human: 59.92, measured on a small validation subset
- Pronouns appear in 53.9% of QuAC questions and in 75.0% of conversation histories. Mixed-category pronouns in the history affect 27.8% of the data, and only about a third of rewrites are reachable by pronoun replacement alone.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/)

## Cited by

Cited by

94 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 92 not yet.

- [Open-Domain Question Answering Goes Conversational via Question Rewriting](https://md.research.elamur.ai/free/en/papers/anantha-2021-open-domain-question-answering-goes/)
- [Bridging Background Knowledge Gaps in Translation with Automatic Explicitation](https://md.research.elamur.ai/free/en/papers/han-2023-bridging-background-knowledge-gaps-translation/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{elgohary2019can,
  title = {{Can You Unpack That? Learning to Rewrite Questions-in-Context}},
  author = {Elgohary, Ahmed and Peskov, Denis and Boyd-Graber, Jordan},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1605},
  url = {https://doi.org/10.18653/v1/d19-1605}
}
```

- [← previous CamemBERT: a Tasty French Language Model](https://md.research.elamur.ai/free/en/papers/martin-2019-camembert-tasty-french-language-model/)
- [next → CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB](https://md.research.elamur.ai/free/en/papers/schwenk-2019-ccmatrix-mining-billions-high-quality/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/)

## References

28 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2015%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+In+Proceedings+of+the+International+Conference+on+Learning+Representati) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
2. Christian Buck, Jannis Bulian, Massimiliano Ciaramita, Wojciech Gajewski, Andrea Gesmundo, Neil Houlsby, and Wei Wang. 2018. Ask the right questions: Active question reformulation with reinforcement learning. In Proceedings of the International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Christian+Buck%2C+Jannis+Bulian%2C+Massimiliano+Ciaramita%2C+Wojciech+Gajewski%2C+Andrea+Gesmundo%2C+Neil+Houlsby%2C+and+Wei+Wang%2E+2018%2E+Ask+the+right+questions%3A+Active+question+reformulation+with+reinforcement+l)
3. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. QuAC: Question answering in context. In Proceedings of Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Eunsol+Choi%2C+He+He%2C+Mohit+Iyyer%2C+Mark+Yatskar%2C+Wen+tau+Yih%2C+Yejin+Choi%2C+Percy+Liang%2C+and+Luke+Zettlemoyer%2E+2018%2E+QuAC%3A+Question+answering+in+context%2E+In+Proceedings+of+Empirical+Methods+in+Natural+Lan) [our card](https://md.research.elamur.ai/free/en/papers/choi-2018-quac-question-answering-context/)
4. Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. 2017. Visual dialog. In Computer Vision and Pattern Recognition. [find](https://scholar.google.com/scholar?q=Abhishek+Das%2C+Satwik+Kottur%2C+Khushi+Gupta%2C+Avi+Singh%2C+Deshraj+Yadav%2C+Jos%C3%A9+MF+Moura%2C+Devi+Parikh%2C+and+Dhruv+Batra%2E+2017%2E+Visual+dialog%2E+In+Computer+Vision+and+Pattern+Recognition%2E)
5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2018%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Conference+of+the+North+American+Chapter+of)
6. Ye Liu, Chenwei Zhang, Xiaohui Yan, Yi Chang, and Philip S Yu. 2019. Generative question refinement with deep reinforcement learning in retrieval-based QA system. In Proceedings of the ACM International Conference on Information and Knowledge Management. [find](https://scholar.google.com/scholar?q=Ye+Liu%2C+Chenwei+Zhang%2C+Xiaohui+Yan%2C+Yi+Chang%2C+and+Philip+S+Yu%2E+2019%2E+Generative+question+refinement+with+deep+reinforcement+learning+in+retrieval%2Dbased+QA+system%2E+In+Proceedings+of+the+ACM+Internation)
7. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Jeffrey+Pennington%2C+Richard+Socher%2C+and+Christopher+Manning%2E+2014%2E+GloVe%3A+Global+vectors+for+word+representation%2E+In+Proceedings+of+Empirical+Methods+in+Natural+Language+Processing%2E)
8. Denis Peskov, Joe Barrow, Pedro Rodriguez, Graham Neubig, and Jordan Boyd-Graber. 2019. Mitigating noisy inputs for question answering. In Proceedings of the Annual Conference of the International Speech Communication Association. [find](https://scholar.google.com/scholar?q=Denis+Peskov%2C+Joe+Barrow%2C+Pedro+Rodriguez%2C+Graham+Neubig%2C+and+Jordan+Boyd%2DGraber%2E+2019%2E+Mitigating+noisy+inputs+for+question+answering%2E+In+Proceedings+of+the+Annual+Conference+of+the+International+Spe)
9. Chen Qu, Liu Yang, Minghui Qiu, W. Bruce Croft, Yongfeng Zhang, and Mohit Iyyer. 2019a. BERT with history modeling for conversational question answering. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. [find](https://scholar.google.com/scholar?q=Chen+Qu%2C+Liu+Yang%2C+Minghui+Qiu%2C+W%2E+Bruce+Croft%2C+Yongfeng+Zhang%2C+and+Mohit+Iyyer%2E+2019a%2E+BERT+with+history+modeling+for+conversational+question+answering%2E+In+Proceedings+of+the+ACM+SIGIR+Conference+on+)
10. Chen Qu, Liu Yang, Minghui Qiu, Yongfeng Zhang, Cen Chen, W. Bruce Croft, and Mohit Iyyer. 2019b. Attentive history selection for conversational question answering. In Proceedings of the ACM International Conference on Information and Knowledge Management. [find](https://scholar.google.com/scholar?q=Chen+Qu%2C+Liu+Yang%2C+Minghui+Qiu%2C+Yongfeng+Zhang%2C+Cen+Chen%2C+W%2E+Bruce+Croft%2C+and+Mohit+Iyyer%2E+2019b%2E+Attentive+history+selection+for+conversational+question+answering%2E+In+Proceedings+of+the+ACM+Internati)
11. Li Dong, Jonathan Mallinson, Siva Reddy, and Mirella Lapata. 2017. Learning to paraphrase for question answering. In Proceedings of Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Li+Dong%2C+Jonathan+Mallinson%2C+Siva+Reddy%2C+and+Mirella+Lapata%2E+2017%2E+Learning+to+paraphrase+for+question+answering%2E+In+Proceedings+of+Empirical+Methods+in+Natural+Language+Processing%2E) [our card](https://md.research.elamur.ai/free/en/papers/dong-2017-learning-paraphrase-question-answering/)
12. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+2016%2E+SQuAD%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+In+Proceedings+of+Empirical+Methods+in+Natural+Language+Processing) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
13. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Conference of the North American Chapter of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Dheeru+Dua%2C+Yizhong+Wang%2C+Pradeep+Dasigi%2C+Gabriel+Stanovsky%2C+Sameer+Singh%2C+and+Matt+Gardner%2E+2019%2E+DROP%3A+A+reading+comprehension+benchmark+requiring+discrete+reasoning+over+paragraphs%2E+In+Conference+o) [our card](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/)
14. Pushpendre Rastogi, Arpit Gupta, Tongfei Chen, and Lambert Mathias. 2019. Scaling multi-domain dialogue state tracking via query reformulation. In Conference of the North American Chapter of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Pushpendre+Rastogi%2C+Arpit+Gupta%2C+Tongfei+Chen%2C+and+Lambert+Mathias%2E+2019%2E+Scaling+multi%2Ddomain+dialogue+state+tracking+via+query+reformulation%2E+In+Conference+of+the+North+American+Chapter+of+the+Assoc)
15. Ahmed Elgohary, Chen Zhao, and Jordan Boyd-Graber. 2018. Dataset and baselines for sequential open-domain question answering. In Proceedings of Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Ahmed+Elgohary%2C+Chen+Zhao%2C+and+Jordan+Boyd%2DGraber%2E+2018%2E+Dataset+and+baselines+for+sequential+open%2Ddomain+question+answering%2E+In+Proceedings+of+Empirical+Methods+in+Natural+Language+Processing%2E)
16. Shi Feng and Jordan Boyd-Graber. 2019. What AI can do for me: Evaluating machine learning interpretations in cooperative play. In International Conference on Intelligent User Interfaces. [find](https://scholar.google.com/scholar?q=Shi+Feng+and+Jordan+Boyd%2DGraber%2E+2019%2E+What+AI+can+do+for+me%3A+Evaluating+machine+learning+interpretations+in+cooperative+play%2E+In+International+Conference+on+Intelligent+User+Interfaces%2E)
17. Hsin-Yuan Huang, Eunsol Choi, and Wen tau Yih. 2019. FlowQA: Grasping flow in history for conversational machine comprehension. In Proceedings of the International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Hsin%2DYuan+Huang%2C+Eunsol+Choi%2C+and+Wen+tau+Yih%2E+2019%2E+FlowQA%3A+Grasping+flow+in+history+for+conversational+machine+comprehension%2E+In+Proceedings+of+the+International+Conference+on+Learning+Representatio)
18. Guillaume Klein, Yoon Kim, Yuntian Deng, Vincent Nguyen, Jean Senellart, and Alexander Rush. 2018. OpenNMT: Neural machine translation toolkit. In Proceedings of Association for Machine Translation in the Americas. [find](https://scholar.google.com/scholar?q=Guillaume+Klein%2C+Yoon+Kim%2C+Yuntian+Deng%2C+Vincent+Nguyen%2C+Jean+Senellart%2C+and+Alexander+Rush%2E+2018%2E+OpenNMT%3A+Neural+machine+translation+toolkit%2E+In+Proceedings+of+Association+for+Machine+Translation+in)
19. Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Siva+Reddy%2C+Danqi+Chen%2C+and+Christopher+D+Manning%2E+2019%2E+CoQA%3A+A+conversational+question+answering+challenge%2E+Transactions+of+the+Association+for+Computational+Linguistics%2E) [our card](https://md.research.elamur.ai/free/en/papers/reddy-2019-coqa-conversational-question-answering-challenge/)
20. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Abigail+See%2C+Peter+J%2E+Liu%2C+and+Christopher+D%2E+Manning%2E+2017%2E+Get+to+the+point%3A+Summarization+with+pointer%2Dgenerator+networks%2E+In+Proceedings+of+the+Association+for+Computational+Linguistics%2E)
21. Rico Sennrich, Orhan Firat, Kyunghyun Cho, Alexandra Birch, Barry Haddow, Julian Hitschler, Marcin Junczys-Dowmunt, Samuel Läubli, Antonio Valerio Miceli Barone, Jozef Mokry, and Maria Nadejde. 2017. Nematus: a toolkit for neural machine translation. In Proceedings of the Software Demonstrations of… [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Orhan+Firat%2C+Kyunghyun+Cho%2C+Alexandra+Birch%2C+Barry+Haddow%2C+Julian+Hitschler%2C+Marcin+Junczys%2DDowmunt%2C+Samuel+L%C3%A4ubli%2C+Antonio+Valerio+Miceli+Barone%2C+Jozef+Mokry%2C+and+Maria+Nadejde%2E+2017%2E+) [our card](https://md.research.elamur.ai/free/en/papers/sennrich-2017-nematus-toolkit-neural-machine-translation/)
22. Hui Su, Xiaoyu Shen, Rongzhi Zhang, Fei Sun, Pengwei Hu, Cheng Niu, and Jie Zhou. 2019. Improving multi-turn dialogue modelling with utterance ReWriter. In Proceedings of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Hui+Su%2C+Xiaoyu+Shen%2C+Rongzhi+Zhang%2C+Fei+Sun%2C+Pengwei+Hu%2C+Cheng+Niu%2C+and+Jie+Zhou%2E+2019%2E+Improving+multi%2Dturn+dialogue+modelling+with+utterance+ReWriter%2E+In+Proceedings+of+the+Association+for+Computati)
23. Saku Sugawara, Kentaro Inui, Satoshi Sekine, and Akiko Aizawa. 2018. What makes reading comprehension questions easier? In Proceedings of Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Saku+Sugawara%2C+Kentaro+Inui%2C+Satoshi+Sekine%2C+and+Akiko+Aizawa%2E+2018%2E+What+makes+reading+comprehension+questions+easier%3F+In+Proceedings+of+Empirical+Methods+in+Natural+Language+Processing%2E)
24. Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. 2019. DREAM: A challenge dataset and models for dialogue-based reading comprehension. Transactions of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kai+Sun%2C+Dian+Yu%2C+Jianshu+Chen%2C+Dong+Yu%2C+Yejin+Choi%2C+and+Claire+Cardie%2E+2019%2E+DREAM%3A+A+challenge+dataset+and+models+for+dialogue%2Dbased+reading+comprehension%2E+Transactions+of+the+Association+for+Comput)
25. Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-base for answering complex questions. In Conference of the North American Chapter of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Alon+Talmor+and+Jonathan+Berant%2E+2018%2E+The+web+as+a+knowledge%2Dbase+for+answering+complex+questions%2E+In+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics%2E)
26. Eric Wallace, Pedro Rodriguez, Shi Feng, Ikuya Yamada, and Jordan Boyd-Graber. 2019. Trick me if you can: Human-in-the-loop generation of adversarial question answering examples. In Transactions of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Eric+Wallace%2C+Pedro+Rodriguez%2C+Shi+Feng%2C+Ikuya+Yamada%2C+and+Jordan+Boyd%2DGraber%2E+2019%2E+Trick+me+if+you+can%3A+Human%2Din%2Dthe%2Dloop+generation+of+adversarial+question+answering+examples%2E+In+Transactions+of+th)
27. Bonnie Webber. 1992. Question answering. In Stuart C. Shapiro, editor, Encyclopedia of Artificial Intelligence, 2nd edition, pages 814–822. John Wiley & Sons, Inc., New York, NY. [find](https://scholar.google.com/scholar?q=Bonnie+Webber%2E+1992%2E+Question+answering%2E+In+Stuart+C%2E+Shapiro%2C+editor%2C+Encyclopedia+of+Artificial+Intelligence%2C+2nd+edition%2C+pages+814%E2%80%93822%2E+John+Wiley+%26+Sons%2C+Inc%2E%2C+New+York%2C+NY%2E)
28. Dani Yogatama, Cyprien de Masson d’Autume, Jerome Connor, Tomás Kociský, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, and Phil Blunsom. 2019. Learning and evaluating general linguistic intelligence. arXiv preprint arXiv:1901.11373. [find](https://scholar.google.com/scholar?q=Dani+Yogatama%2C+Cyprien+de+Masson+d%E2%80%99Autume%2C+Jerome+Connor%2C+Tom%C3%A1s+Kocisk%C3%BD%2C+Mike+Chrzanowski%2C+Lingpeng+Kong%2C+Angeliki+Lazaridou%2C+Wang+Ling%2C+Lei+Yu%2C+Chris+Dyer%2C+and+Phil+Blunsom%2E+2019%2E+Learning+and+evalua)