---
title: "The Impact of Multiple Parallel Phrase Suggestions on Email"
description: "The authors built a browser-based email editor that offers GPT-2 phrase continuations and refined it… 74 references, cited by 12 papers here."
authors: ["Daniel Buschek", "Martin Zurn", "Malin Eiband"]
year: 2021
venue: "arXiv:2101.09157"
source: "https://arxiv.org/abs/2101.09157"
doi: "10.1145/3411764.3445372"
arxiv: "2101.09157"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/buschek-2021-impact-multiple-parallel-phrase-suggestions/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/buschek-2021-impact-multiple-parallel-phrase-suggestions/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) / The Impact of Multiple Parallel Phrase Suggestions on Email Input and Composition Behaviour of Native and Non-Native English Writers

# The Impact of Multiple Parallel Phrase Suggestions on Email Input and Composition Behaviour of Native and Non-Native English Writers

**Authors:** Daniel Buschek, Martin Zurn and Malin Eiband · **Year:** 2021 · **Venue:** arXiv:2101.09157 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://doi.org/10.1145/3411764.3445372) · [PDF](https://arxiv.org/pdf/2101.09157) · [DOI 10.1145/3411764.3445372](https://doi.org/10.1145/3411764.3445372) · [arXiv 2101.09157](https://arxiv.org/abs/2101.09157) · [OpenAlex](https://openalex.org/W3123978464)

**Citations:** 106 citations (OpenAlex, 2026-10-14) · [12](https://md.research.elamur.ai/free/en/papers/buschek-2021-impact-multiple-parallel-phrase-suggestions/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Predictive text tools are usually judged by how many keystrokes or seconds they save. Multi-word suggestions, however, may also give writers ideas about content and phrasing. This paper asks how the number of phrase suggestions shown at once affects typing and composition, and whether native and non-native English writers respond differently. The authors built a browser-based email editor that offers GPT-2 phrase continuations and refined it in a prestudy with 30 participants. They then ran an online study with 156 people, who wrote emails under four conditions showing 0, 1, 3 or 6 parallel suggestions. Interaction logs and participant feedback were analysed. Showing several phrases helped ideation (deciding what to say next and how to say it) but cost efficiency. Non-native writers benefited more from longer suggestion lists. The analysis also identifies nine recurring behaviour patterns covering typing, navigation, revision and suggestion use. The work matters because interfaces tuned only for speed can hide real benefits for less proficient writers. The prototype, model, data and study materials are openly released.

## Key points
- The suggestion engine is GPT-2 inside a React email web app. Users can pick suggestions by mouse or keyboard: tab takes the top one, arrow keys move through the list, and enter accepts.
- Design: a prestudy (N=30) shaped the prototype, followed by an online study (N=156) with four within-person conditions (0/1/3/6 suggestions) and four emails per participant, roughly 20 minutes of use.
- Participants got faster across tasks. Mean writing time fell from 260 s to 181 s between the first and fourth email, and mean suggestion selection time fell from 4309 ms to 3656 ms.
- 85 % of participants said they use suggestions at least sometimes. Multiple parallel phrase suggestions were probably new to most of them.
- Despite fine-tuning, quality was imperfect. Participants mentioned problems with grammar and with topic or formality, and at least two saw inappropriate words. Materials are released at osf.io/7q4c8.

## Interesting ideas
- Treat "how many alternatives to show" as an explicit experimental factor, including a zero baseline, instead of fixing it by screen space; this exposes a real trade-off between speed and inspiration.
- Split evaluation by user proficiency, because a design that looks neutral on average can quietly disadvantage one group.
- The authors argue that bias can sit in the UI policy, not only in the model, for example by gating suggestions on estimated keystroke savings.
- Mine detailed interaction logs for recurring sequential behaviour patterns, such as when people revise or accept text, rather than reporting only end metrics.
- Prefer free composition tasks over copy tasks for external validity, and track learning effects across repeated tasks.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (answer): When a reader returns candidate answers or readings, we could compare showing 0, 1, 3 or 6 ranked options with explanations, and measure whether more alternatives help users judge correctness or only slow them down.
- **I2** (learn): Evaluation runs could report results separately for texts by native and non-native writers, so that regressions hidden in aggregate scores show up.
- **I3** (back): Generation choices such as producing only the single most probable surface form should be audited for whom they serve, perhaps by offering paraphrase variants from the meaning graph when users need wording help.
- **I4** (learn): Logs of how the repair rules and witness parsers disagree over time could be clustered into recurring patterns, explaining where the pipeline revises itself.
- **I5** (answer): Test sets should include freely written texts, not only curated ones, and evaluations should check for order or familiarity effects across repeated runs.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Ludwig-Maximilians-Universität München](https://md.research.elamur.ai/free/en/universities/ludwig-maximilians-universitat-munchen/) — University, Munich, Germany · 44 works · 55,193 citations

Also on the byline: [University of Bayreuth](https://ror.org/0234wmv40).

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🖱️ Human-Computer Interaction](https://md.research.elamur.ai/free/en/shelves/computer-science/human-computer-interaction/) · 61 works

🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 1,373 works

## Cite

Cite as: Daniel Buschek, Martin Zurn and Malin Eiband. 2021. The Impact of Multiple Parallel Phrase Suggestions on Email Input and Composition Behaviour of Native and Non-Native English Writers. arXiv:2101.09157.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{buschek2021impact,
  title = {{The Impact of Multiple Parallel Phrase Suggestions on Email Input and Composition Behaviour of Native and Non-Native English Writers}},
  author = {Daniel Buschek, Martin Zurn and Malin Eiband},
  year = {2021},
  doi = {10.1145/3411764.3445372},
  eprint = {2101.09157},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1145/3411764.3445372}
}
```

- [← previous Human-in-the-Loop for Data Collection: a Multi-Target Counter Narrative Dataset to Fight Online Hate Speech](https://md.research.elamur.ai/free/en/papers/fanton-2021-human-loop-data-collection-multi/)
- [next → Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task](https://md.research.elamur.ai/free/en/papers/tang-2021-improving-speech-translation-understanding-learning/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/)

## References

74 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ahmed Sabbir Arif, Cristina Sylla, and Ali Mazalek. 2016. Learning New Words and Spelling with Autocorrections. In Proceedings of the 2016 ACM International Conference on Interactive Surfaces and Spaces (Niagara Falls, Ontario, Canada) (ISS ’16). Association for Computing Machinery, New York, NY, USA, 409–414. https://doi.org/10.1145/2992154.2996790 [↗](https://doi.org/10.1145/2992154.2996790)
2. Kenneth C. Arnold, Kai-Wei Chang, and Adam T. Kalai. 2017. Counterfactual Language Model Adaptation for Suggesting Phrases. arXiv:1710.01799 [cs.CL] [↗](https://arxiv.org/abs/1710.01799)
3. Kenneth C. Arnold, Krysta Chauncey, and Krzysztof Z. Gajos. 2018. Sentiment Bias in Predictive Text Recommendations Results in Biased Writing. In Proceedings of the 44th Graphics Interface Conference (Toronto, Canada) (GI ’18). Canadian Human-Computer Communications Society, Waterloo, CAN, 42–49. https://doi.org/10.20380/GI2018.07 [↗](https://doi.org/10.20380/gi2018.07)
4. Kenneth C. Arnold, Krysta Chauncey, and Krzysztof Z. Gajos. 2020. Predictive Text Encourages Predictable Writing. In Proceedings of the 25th International Conference on Intelligent User Interfaces (Cagliari, Italy) (IUI ’20). Association for Computing Machinery, New York, NY, USA, 128–138. https://doi.org/10.1145/3377325.3377523 [↗](https://doi.org/10.1145/3377325.3377523)
5. Kenneth C. Arnold, Krzysztof Z. Gajos, and Adam T. Kalai. 2016. On Suggesting Phrases vs. Predicting Words for Mobile Text Composition. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (Tokyo, Japan) (UIST ’16). Association for Computing Machinery, New York, NY, USA, 603–608. https://doi.org/10.1145/2984511.2984584 [↗](https://doi.org/10.1145/2984511.2984584)
6. Gilles Bailly, Antti Oulasvirta, Duncan P. Brumby, and Andrew Howes. 2014. Model of Visual Search and Selection Time in Linear Menus. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI ’14). Association for Computing Machinery, New York, NY, USA, 3865–3874. https://doi.org/10.1145/2556288.2557093 [↗](https://doi.org/10.1145/2556288.2557093)
7. Nikola Banovic, Ticha Sethapakdi, Yasasvi Hari, Anind K. Dey, and Jennifer Mankoff. 2019. The Limits of Expert Text Entry Speed on Mobile Keyboards with Autocorrect. In Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services (Taipei, Taiwan) (MobileHCI ’19). Association for Computing Machinery, New York, NY, USA, Article 15, 12 pages. https://doi.org/10.1145/3338286.3340126 [↗](https://doi.org/10.1145/3338286.3340126)
8. David Barber. 2012. Bayesian Rreasoning and Machine Learning. Cambridge University Press. [http://web4.cs.ucl.ac.uk/staff/D.Barber/textbook/090310.pdf](http://web4.cs.ucl.ac.uk/staff/D.Barber/textbook/090310.pdf) [↗](http://web4.cs.ucl.ac.uk/staff/D.Barber/textbook/090310.pdf)
9. Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. 2015. [Fitting Linear Mixed-Effects Models Using lme4](https://md.research.elamur.ai/free/en/papers/bates-2015-fitting-linear-mixed-effects-models/). Journal of Statistical Software 67, 1 (2015), 1–48. https://doi.org/10.18637/jss.v067.i01 [↗](https://doi.org/10.18637/jss.v067.i01)
10. Xiaojun Bi, Tom Ouyang, and Shumin Zhai. 2014. Both Complete and Correct? Multi-Objective Optimization of Touchscreen Keyboard. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI ’14). Association for Computing Machinery, New York, NY, USA, 2297–2306. https://doi.org/10.1145/2556288.2557414 [↗](https://doi.org/10.1145/2556288.2557414)
11. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. [Language (Technology) is Power: A Critical Survey of “Bias” in NLP](https://md.research.elamur.ai/free/en/papers/blodgett-2020-language-technology-is-power-critical/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 5454–5476. https://doi.org/10.18653/v1/2020.acl-main.485 [↗](https://doi.org/10.18653/v1/2020.acl-main.485)
12. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). arXiv:2005.14165 [cs.CL] [↗](https://arxiv.org/abs/2005.14165)
13. Heloisa Candello, Cosmin Munteanu, Leigh Clark, Jaisie Sin, María Inés Torres, Martin Porcheron, Chelsea M. Myers, Benjamin Cowan, Joel Fischer, Stephan Schlögl, Christine Murad, and Stuart Reeves. 2020. CUI@CHI: Mapping Grand Challenges for the Conversational User Interface Community. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI EA ’20). Association for Computing Machinery, New York, NY, USA, 1–8. https://doi.org/10.1145/3334480.3375152 [↗](https://doi.org/10.1145/3334480.3375152)
14. N. Carr. 2015. The Glass Cage: Where Automation is Taking Us. Random House, London, UK. [find](https://scholar.google.com/scholar?q=N%2E+Carr%2E+2015%2E+The+Glass+Cage%3A+Where+Automation+is+Taking+Us%2E+Random+House%2C+London%2C+UK%2E)
15. Gilles Celeux and Jean-Baptiste Durand. 2008. Selecting hidden Markov model state number with cross-validated likelihood. Computational Statistics 23, 4 (Oct. 2008), 541–564. https://doi.org/10.1007/s00180-007-0097-1 [↗](https://doi.org/10.1007/s00180-007-0097-1)
16. Mia Xu Chen, Benjamin N. Lee, Gagan Bansal, Yuan Cao, Shuyuan Zhang, Justin Lu, Jackie Tsay, Yinan Wang, Andrew M. Dai, Zhifeng Chen, Timothy Sohn, and Yonghui Wu. 2019. Gmail Smart Compose: Real-Time Assisted Writing. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (Anchorage, AK, USA) (KDD ’19). Association for Computing Machinery, New York, NY, USA, 2287–2295. https://doi.org/10.1145/3292500.3330723 [↗](https://doi.org/10.1145/3292500.3330723)
17. Elizabeth Clark, Anne Spencer Ross, Chenhao Tan, Yangfeng Ji, and Noah A. Smith. 2018. Creative Writing with a Machine in the Loop: Case Studies on Slogans and Stories. In 23rd International Conference on Intelligent User Interfaces (Tokyo, Japan) (IUI ’18). Association for Computing Machinery, New York, NY, USA, 329–340. https://doi.org/10.1145/3172944.3172983 [↗](https://doi.org/10.1145/3172944.3172983)
18. James Clawson, Ahmed Sabbir Arif, Stephen Brewster, Mark Dunlop, Per Ola Kristensson, and Antti Oulasvirta. 2015. Text Entry on the Edge. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI EA ’15). Association for Computing Machinery, New York, NY, USA, 2381–2384. https://doi.org/10.1145/2702613.2702660 [↗](https://doi.org/10.1145/2702613.2702660)
19. Paul Deane, Nora Odendahl, Thomas Quinlan, Mary Fowles, Cyndi Welsh, and Jennifer Bivens-Tatum. 2008. Cognitive models of writing: Writing proficiency as a complex integrated skill. ETS Research Report Series 2008, 2 (2008), i–36. https://doi.org/10.1002/j.2333-8504.2008.tb02141.x arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/j.2333-8504.2008.tb02141.x [↗](https://doi.org/10.1002/j.2333-8504.2008.tb02141.x)
20. Budhaditya Deb, Peter Bailey, and Milad Shokouhi. 2019. Diversifying Reply Suggestions using a Matching-Conditional Variational Autoencoder. arXiv:1903.10630 [cs.CL] [↗](https://arxiv.org/abs/1903.10630)
21. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186. https://doi.org/10.18653/v1/n19-1423 [↗](https://doi.org/10.18653/v1/n19-1423)
22. Mark Dunlop and John Levine. 2012. Multidimensional Pareto Optimization of Touchscreen Keyboards for Speed, Familiarity and Improved Spell Checking. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Austin, Texas, USA) (CHI ’12). Association for Computing Machinery, New York, NY, USA, 2669–2678. https://doi.org/10.1145/2207676.2208659 [↗](https://doi.org/10.1145/2207676.2208659)
23. Morgan Magnus Fleming. 2019. Tools That Teach Too: Exploring the Role of Interaction in the Development of Useful Cognitive Residuals. Ph.D. Dissertation. UC Merced. [find](https://scholar.google.com/scholar?q=Morgan+Magnus+Fleming%2E+2019%2E+Tools+That+Teach+Too%3A+Exploring+the+Role+of+Interaction+in+the+Development+of+Useful+Cognitive+Residuals%2E+Ph%2ED%2E+Dissertation%2E+UC+Merced%2E)
24. Linda Flower and John R. Hayes. 1981. A Cognitive Process Theory of Writing. College Composition and Communication 32, 4 (1981), 365–387. http://www.jstor.org/stable/356600 [↗](https://doi.org/10.2307/356600)
25. Andrew Fowler, Kurt Partridge, Ciprian Chelba, Xiaojun Bi, Tom Ouyang, and Shumin Zhai. 2015. Effects of Language Modeling and Its Personalization on Touchscreen Typing Performance. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI ’15). Association for Computing Machinery, New York, NY, USA, 649–658. https://doi.org/10.1145/2702123.2702503 [↗](https://doi.org/10.1145/2702123.2702503)
26. Jonas Frich, Lindsay MacDonald Vermeulen, Christian Remy, Michael Mose Biskjaer, and Peter Dalsgaard. 2019. Mapping the Landscape of Creativity Support Tools in HCI. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–18. https://doi.org/10.1145/3290605.3300619 [↗](https://doi.org/10.1145/3290605.3300619)
27. S. Gehrmann, H. Strobelt, R. Krüger, H. Pfister, and A. M. Rush. 2020. Visual Interaction with Deep Learning Models through Collaborative Semantic Inference. IEEE Transactions on Visualization and Computer Graphics 26, 1 (2020), 884–894. [↗](https://doi.org/10.1109/tvcg.2019.2934595)
28. Katy Ilonka Gero and Lydia B. Chilton. 2019. Metaphoria: An Algorithmic Companion for Metaphor Creation. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300526 [↗](https://doi.org/10.1145/3290605.3300526)
29. Marjan Ghazvininejad, Xing Shi, Jay Priyadarshi, and Kevin Knight. 2017. [Hafez: an Interactive Poetry Generation System](https://md.research.elamur.ai/free/en/papers/ghazvininejad-2017-hafez-interactive-poetry-generation-system/). In Proceedings of ACL 2017, System Demonstrations. Association for Computational Linguistics, Vancouver, Canada, 43–48. https://www.aclweb.org/anthology/P17-4008 [↗](https://aclanthology.org/P17-4008/)
30. Mayank Goel, Leah Findlater, and Jacob Wobbrock. 2012. WalkType: Using Accelerometer Data to Accomodate Situational Impairments in Mobile Touch Screen Text Entry. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Austin, Texas, USA) (CHI ’12). Association for Computing Machinery, New York, NY, USA, 2687–2696. https://doi.org/10.1145/2207676.2208662 [↗](https://doi.org/10.1145/2207676.2208662)
31. Joshua Goodman, Gina Venolia, Keith Steury, and Chauncey Parker. 2002. Language Modeling for Soft Keyboards. In Proceedings of the 7th International Conference on Intelligent User Interfaces (San Francisco, California, USA) (IUI ’02). Association for Computing Machinery, New York, NY, USA, 194–195. https://doi.org/10.1145/502716.502753 [↗](https://doi.org/10.1145/502716.502753)
32. Mitchell Gordon, Tom Ouyang, and Shumin Zhai. 2016. WatchWriter: Tap and Gesture Typing on a Smartwatch Miniature Keyboard with Statistical Decoding. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 3817–3821. https://doi.org/10.1145/2858036.2858242 [↗](https://doi.org/10.1145/2858036.2858242)
33. Jay Henderson, Sylvain Malacria, Mathieu Nancel, and Edward Lank. 2020. Investigating the Necessity of Delay in Marking Menu Invocation. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376296 [↗](https://doi.org/10.1145/3313831.3376296)
34. Ray Hyman. 1953. Stimulus information as a determinant of reaction time. Journal of experimental psychology 45, 3 (1953), 188. [↗](https://doi.org/10.1037/h0056940)
35. Christina James and Michael Longé. 2000. Bringing Text Input beyond the Desktop. In CHI ’00 Extended Abstracts on Human Factors in Computing Systems (The Hague, The Netherlands) (CHI EA ’00). Association for Computing Machinery, New York, NY, USA, 49–50. https://doi.org/10.1145/633292.633324 [↗](https://doi.org/10.1145/633292.633324)
36. Anjuli Kannan, Karol Kurach, Sujith Ravi, Tobias Kaufmann, Andrew Tomkins, Balint Miklos, Greg Corrado, Laszlo Lukacs, Marina Ganea, Peter Young, and Vivek Ramavajjala. 2016. Smart Reply: Automated Response Suggestion for Email. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (San Francisco, California, USA) (KDD ’16). Association for Computing Machinery, New York, NY, USA, 955–964. https://doi.org/10.1145/2939672.2939801 [↗](https://doi.org/10.1145/2939672.2939801)
37. Bryan Klimt and Yiming Yang. 2004. The Enron Corpus: A New Dataset for Email Classification Research. In Machine Learning: ECML 2004, Jean-François Boulicaut, Floriana Esposito, Fosca Giannotti, and Dino Pedreschi (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 217–226. [find](https://scholar.google.com/scholar?q=Bryan+Klimt+and+Yiming+Yang%2E+2004%2E+The+Enron+Corpus%3A+A+New+Dataset+for+Email+Classification+Research%2E+In+Machine+Learning%3A+ECML+2004%2C+Jean%2DFran%C3%A7ois+Boulicaut%2C+Floriana+Esposito%2C+Fosca+Giannotti%2C+and+D)
38. Andreas Komninos, Mark Dunlop, Kyriakos Katsaris, and John Garofalakis. 2018. A Glimpse of Mobile Text Entry Errors and Corrective Behaviour in the Wild. In Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct (Barcelona, Spain) (MobileHCI ’18). Association for Computing Machinery, New York, NY, USA, 221–228. https://doi.org/10.1145/3236112.3236143 [↗](https://doi.org/10.1145/3236112.3236143)
39. Per Ola Kristensson, Stephen Brewster, James Clawson, Mark Dunlop, Leah Findlater, Poika Isokoski, Benoît Martin, Antti Oulasvirta, Keith Vertanen, and Annalu Waller. 2013. Grand Challenges in Text Entry. In CHI ’13 Extended Abstracts on Human Factors in Computing Systems (Paris, France) (CHI EA ’13). Association for Computing Machinery, New York, NY, USA, 3315–3318. https://doi.org/10.1145/2468356.2479675 [↗](https://doi.org/10.1145/2468356.2479675)
40. Per Ola Kristensson and Keith Vertanen. 2014. The Inviscid Text Entry Rate and Its Application as a Grand Goal for Mobile Text Entry. In Proceedings of the 16th International Conference on Human-Computer Interaction with Mobile Devices & Services (Toronto, ON, Canada) (MobileHCI ’14). Association for Computing Machinery, New York, NY, USA, 335–338. https://doi.org/10.1145/2628363.2628405 [↗](https://doi.org/10.1145/2628363.2628405)
41. Alexandra Kuznetsova, Per B. Brockhoff, and Rune H. B. Christensen. 2017. lmerTest Package: Tests in Linear Mixed Effects Models. Journal of Statistical Software 82, 13 (2017), 1–26. https://doi.org/10.18637/jss.v082.i13 [↗](https://doi.org/10.18637/jss.v082.i13)
42. T. K. Landauer and D. W. Nachbar. 1985. Selection from Alphabetic and Numeric Menu Trees Using a Touch Screen: Breadth, Depth, and Width. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (San Francisco, California, USA) (CHI ’85). Association for Computing Machinery, New York, NY, USA, 73–78. https://doi.org/10.1145/317456.317470 [↗](https://doi.org/10.1145/317456.317470)
43. Mariëlle Leijten, Eric Van Horenbeeck, and Luuk Van Waes. 25 Jan. 2019. Analysing Keystroke Logging Data from a Linguistic Perspective. Brill, Leiden, Niederlande, 71 – 95. https://doi.org/10.1163/9789004392526_005 [↗](https://doi.org/10.1163/9789004392526_005)
44. Mariëlle Leijten and Luuk Van Waes. 2013. Keystroke Logging in Writing Research: Using Inputlog to Analyze and Visualize Writing Processes. Written Communication 30, 3 (2013), 358–392. https://doi.org/10.1177/0741088313491692 arXiv:https://doi.org/10.1177/0741088313491692 [↗](https://doi.org/10.1177/0741088313491692)
45. Yang Li, Sayan Sarcar, Sunjun Kim, and Xiangshi Ren. 2020. Swap: A Replacement-Based Text Revision Technique for Mobile Devices. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3313831.3376217 [↗](https://doi.org/10.1145/3313831.3376217)
46. Eva Lindgren and Kirk Sullivan. 2019. Observing Writing. Brill, Leiden, Niederlande. https://doi.org/10.1163/9789004392526 [↗](https://doi.org/10.1163/9789004392526)
47. Cerstin Mahlow and Robert Dale. 2014. 12 Production media: Writing as using tools in media convergent environments. De Gruyter Mouton, Berlin, Boston, 209 – 230. https://doi.org/10.1515/9783110220674.209 [↗](https://doi.org/10.1515/9783110220674.209)
48. Brendan D. McKay, Alison Meynert, and Wendy Myrvold. 2007. Small latin squares, quasigroups, and loops. Journal of Combinatorial Designs 15, 2 (2007), 98–119. https://doi.org/10.1002/jcd.20105 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/jcd.20105 [↗](https://doi.org/10.1002/jcd.20105)
49. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model Cards for Model Reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency (Atlanta, GA, USA) (FAT* ’19). Association for Computing Machinery, New York, NY, USA, 220–229. https://doi.org/10.1145/3287560.3287596 [↗](https://doi.org/10.1145/3287560.3287596)
50. Hugo Nicolau, André Rodrigues, André Santos, Tiago Guerreiro, Kyle Montague, and João Guerreiro. 2019. The Design Space of Nonvisual Word Completion. In The 21st International ACM SIGACCESS Conference on Computers and Accessibility (Pittsburgh, PA, USA) (ASSETS ’19). Association for Computing Machinery, New York, NY, USA, 249–261. https://doi.org/10.1145/3308561.3353786 [↗](https://doi.org/10.1145/3308561.3353786)
51. Cyrus Omar, Young Seok Yoon, Thomas D LaToza, and Brad A Myers. 2012. Active code completion. In 2012 34th International Conference on Software Engineering (ICSE). IEEE, 859–869. https://doi.org/10.1109/ICSE.2012.6227133 [↗](https://doi.org/10.1109/icse.2012.6227133)
52. Kseniia Palin, Anna Maria Feit, Sunjun Kim, Per Ola Kristensson, and Antti Oulasvirta. 2019. How Do People Type on Mobile Devices? Observations from a Study with 37,000 Volunteers. In Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services (Taipei, Taiwan) (MobileHCI ’19). Association for Computing Machinery, New York, NY, USA, Article 9, 12 pages. https://doi.org/10.1145/3338286.3340120 [↗](https://doi.org/10.1145/3338286.3340120)
53. Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. 2020. [Pre-trained Models for Natural Language Processing: A Survey](https://md.research.elamur.ai/free/en/papers/qiu-2020-pre-trained-models-natural-language/). arXiv:2003.08271 [cs.CL] [↗](https://arxiv.org/abs/2003.08271)
54. Philip Quinn and Shumin Zhai. 2016. A Cost-Benefit Study of Text Entry Suggestion Interaction. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 83–88. https://doi.org/10.1145/2858036.2858305 [↗](https://doi.org/10.1145/2858036.2858305)
55. R Core Team. 2020. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. [https://www.R-project.org](https://www.R-project.org) [↗](https://www.R-project.org)
56. L. Rabiner. 1989. A tutorial on hidden Markov models and selected applications in speech recognition. Proc. IEEE 77, 2 (Feb 1989), 257–286. [↗](https://doi.org/10.1109/5.18626)
57. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+Blog+1%2C+8+%282019%29%2C+9%2E)
58. R. Robbes and M. Lanza. 2008. How Program History Can Improve Code Completion. In Proceedings of the 2008 23rd IEEE/ACM International Conference on Automated Software Engineering (ASE ’08). IEEE Computer Society, USA, 317–326. https://doi.org/10.1109/ASE.2008.42 [↗](https://doi.org/10.1109/ase.2008.42)
59. Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, Miles McCain, Alex Newhouse, Jason Blazakis, Kris McGuffie, and Jasmine Wang. 2019. [Release Strategies and the Social Impacts of Language Models](https://md.research.elamur.ai/free/en/papers/solaiman-2019-release-strategies-social-impacts-language/). arXiv:1908.09203 [cs.CL] [↗](https://arxiv.org/abs/1908.09203)
60. Carola Strobl, Emilie Ailhaud, Kalliopi Benetos, Ann Devitt, Otto Kruse, Antje Proske, and Christian Rapp. 2019. Digital support for academic writing: A review of technologies and pedagogies. Computers & Education 131 (2019), 33 – 48. https://doi.org/10.1016/j.compedu.2018.12.005 [↗](https://doi.org/10.1016/j.compedu.2018.12.005)
61. Pradyumna Tambwekar, Murtaza Dhuliawala, Lara J. Martin, Animesh Mehta, Brent Harrison, and Mark O. Riedl. 2018. Controllable Neural Story Plot Generation via Reinforcement Learning. arXiv:1809.10736 [cs.CL] [↗](https://arxiv.org/abs/1809.10736)
62. Maartje ter Hoeve, Robert Sim, Elnaz Nouri, Adam Fourney, Maarten de Rijke, and Ryen W. White. 2020. Conversations with Documents: An Exploration of Document-Centered Assistance. In Proceedings of the 2020 Conference on Human Information Interaction and Retrieval (Vancouver BC, Canada) (CHIIR ’20). Association for Computing Machinery, New York, NY, USA, 43–52. https://doi.org/10.1145/3343413.3377971 [↗](https://doi.org/10.1145/3343413.3377971)
63. Anestis Touloumis. 2015. R Package multgee: A Generalized Estimating Equations Solver for Multinomial Responses. Journal of Statistical Software 64, 8 (2015), 1–14. [http://www.jstatsoft.org/v64/i08/](http://www.jstatsoft.org/v64/i08/) [↗](http://www.jstatsoft.org/v64/i08/)
64. Keith Trnka, John McCaw, Debra Yarrington, Kathleen F. McCoy, and Christopher Pennington. 2009. User Interaction with Word Prediction: The Effects of Prediction Quality. ACM Trans. Access. Comput. 1, 3, Article 17 (Feb. 2009), 34 pages. https://doi.org/10.1145/1497302.1497307 [↗](https://doi.org/10.1145/1497302.1497307)
65. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. [Attention is All you Need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems 30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.). Curran Associates, Inc., 5998–6008. [http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf](http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf) [↗](http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf)
66. Keith Vertanen, Mark Dunlop, James Clawson, Per Ola Kristensson, and Ahmed Sabbir Arif. 2016. Inviscid Text Entry and Beyond. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (San Jose, California, USA) (CHI EA ’16). Association for Computing Machinery, New York, NY, USA, 3469–3476. https://doi.org/10.1145/2851581.2856472 [↗](https://doi.org/10.1145/2851581.2856472)
67. Keith Vertanen and Per Ola Kristensson. 2014. Complementing Text Entry Evaluations with a Composition Task. ACM Trans. Comput.-Hum. Interact. 21, 2, Article 8 (Feb. 2014), 33 pages. https://doi.org/10.1145/2555691 [↗](https://doi.org/10.1145/2555691)
68. Keith Vertanen, Haythem Memmi, Justin Emge, Shyam Reyal, and Per Ola Kristensson. 2015. VelociTap: Investigating Fast Mobile Text Entry Using Sentence-Based Decoding of Touchscreen Keyboard Input. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI ’15). Association for Computing Machinery, New York, NY, USA, 659–668. https://doi.org/10.1145/2702123.2702135 [↗](https://doi.org/10.1145/2702123.2702135)
69. Keith Vertanen, Kyle Montague, Mark Dunlop, Ahmed Sabbir Arif, Xiaojun Bi, and Shiri Azenkot. 2017. Ubiquitous Text Interaction. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI EA ’17). Association for Computing Machinery, New York, NY, USA, 566–573. https://doi.org/10.1145/3027063.3027066 [↗](https://doi.org/10.1145/3027063.3027066)
70. Thomas Wolf. 2019. How to build a State-of-the-Art Conversational AI with Transfer Learning. [https://medium.com/huggingface/how-to-build-a-state-of-the-art-conversational-ai-with-transfer-learning-2d818ac26313](https://medium.com/huggingface/how-to-build-a-state-of-the-art-conversational-ai-with-transfer-learning-2d818ac26313) [↗](https://medium.com/huggingface/how-to-build-a-state-of-the-art-conversational-ai-with-transfer-learning-2d818ac26313)
71. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. HuggingFace’s [Transformers: State-of-the-art Natural Language Processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). arXiv:1910.03771 [cs.CL] [↗](https://arxiv.org/abs/1910.03771)
72. Qian Yang, Justin Cranshaw, Saleema Amershi, Shamsi T. Iqbal, and Jaime Teevan. 2019. Sketching NLP: A Case Study of Exploring the Right Things To Design with Language Intelligence. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300415 [↗](https://doi.org/10.1145/3290605.3300415)
73. Ying Yin, Tom Yu Ouyang, Kurt Partridge, and Shumin Zhai. 2013. Making Touchscreen Keyboards Adaptive to Keys, Hand Postures, and Individuals: A Hierarchical Spatial Backoff Model Approach. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Paris, France) (CHI ’13). Association for Computing Machinery, New York, NY, USA, 2775–2784. https://doi.org/10.1145/2470654.2481384 [↗](https://doi.org/10.1145/2470654.2481384)
74. Mingrui Ray Zhang, He Wen, and Jacob O. Wobbrock. 2019. Type, Then Correct: Intelligent Text Correction Techniques for Mobile Text Entry Using Neural Networks. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (New Orleans, LA, USA) (UIST ’19). Association for Computing Machinery, New York, NY, USA, 843–855. https://doi.org/10.1145/3332165.3347924 [↗](https://doi.org/10.1145/3332165.3347924)

*Updated 24 October 2026*