--- title: "How Useful is Context, Actually? Comparing LLMs and Humans" description: "The paper asks how well language models and people can recover the English discourse marker… 5 references, cited by 4 papers here." authors: ["Emily Sadlier-Brown", "Millie Lou", "Miikka Silfverberg", "Carla Kam"] year: 2024 venue: "Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics" source: "https://aclanthology.org/2024.cmcl-1.20/" doi: "10.18653/v1/2024.cmcl-1.20" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/brown-2024-how-useful-is-context-actually/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/brown-2024-how-useful-is-context-actually/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/10/) / How Useful is Context, Actually? Comparing LLMs and Humans on Discourse Marker Prediction # How Useful is Context, Actually? Comparing LLMs and Humans on Discourse Marker Prediction **Authors:** Emily Sadlier-Brown, Millie Lou, [Miikka Silfverberg](https://md.research.elamur.ai/free/en/authors/miikka-silfverberg/) and Carla Kam · **Year:** 2024 · **Venue:** [Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics](https://md.research.elamur.ai/free/en/venues/cmcl/) ([CMCL 2024](https://md.research.elamur.ai/free/en/venues/cmcl/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2024.cmcl-1.20/) · [PDF](https://aclanthology.org/2024.cmcl-1.20.pdf) · [DOI 10.18653/v1/2024.cmcl-1.20](https://doi.org/10.18653/v1/2024.cmcl-1.20) · [OpenAlex](https://openalex.org/W4402671110) **Citations:** 3 citations (OpenAlex, 2026-10-14) · [4](https://md.research.elamur.ai/free/en/papers/brown-2024-how-useful-is-context-actually/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary The paper asks how well language models and people can recover the English discourse marker *actually* when it is removed from real conversational speech, and how much surrounding talk helps. The authors took 295 adult utterances containing *actually* from the Providence Corpus (PhonBank), which records home interactions around young children. They built cloze items in four context conditions: target alone, target plus next utterance, previous utterance plus target, and all three. Answers were open-ended, not chosen from a list. mTurk workers and five models (BERT, RoBERTa, ELECTRA, GPT-3.5, GPT-4) each produced five guesses, scored by recall@5. Context always helped, but how much, and from which side, varied by reader. Preceding context mattered most for humans and GPT-4, and GPT-4 slightly beat humans overall. On GPT-3.5-generated filler items, several models far outscored humans. This suggests synthetic text favours models and can distort human–model comparisons, so pragmatic evaluation needs natural data and careful context design. ## Key points - Data: 295 natural *actually* utterances from the Providence Corpus, each with one preceding and one following utterance. About 70% are child-directed. The set adds 37 corpus distractors and 295 GPT-3.5 synthetic items, for 2360 cloze items across four conditions (T, T+N, P+T, P+T+N). - Humans: 255 mTurk participants, about 8.1 items each, each giving 5 candidate words. Responses with many repeated answers or words copied from the prompt were filtered out. - On *actually* items, the best recall@5 scores came with full context: GPT-4 34.2, humans 31.5, RoBERTa 25.8, GPT-3.5 22.4, BERT 21.0, ELECTRA 13.6. - Adding the preceding utterance gave GPT-4 roughly +8 points and GPT-3.5 roughly +6 points. RoBERTa gained little from context, yet it beat GPT-3.5 in every condition. - On synthetic items with full context, GPT-4 reached 91.2 versus 51.2 for humans. Every model except ELECTRA beat humans there. A post-hoc check found that people who did more items guessed *actually* somewhat more often, but the effect was moderate. ## Interesting ideas - Testing the same item under target-only, left-context, right-context and both-sides conditions separates how much each side of the discourse contributes, rather than giving one blended score. - The authors let readers produce answers freely instead of picking from a fixed list, because a closed list makes the task easier and inflates scores. - Text written by a model is unusually predictable for models, so a model-generated test set measures fit to that model's distribution more than real competence. - Discourse markers like *actually* signal contrast and expectation through regular cues in nearby utterances, which makes them predictable from the surrounding conversation. - The authors used filler items aimed at other words to keep annotators from learning the hidden target, then checked afterwards whether the target-answer rate rose with exposure. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (frames): Run the prag implicature and speech-act components with and without the previous and next sentence, to measure which pragmatic decisions truly depend on prior discourse. - **I2** (answer): Score the fable and tale QA readers on unconstrained answers as well as multiple choice, so that guessing by elimination cannot hide weak understanding. - **I3** (learn): Keep a natural-text evaluation split apart from any LLM-teacher-generated data, and report the gap between the two as a negative control. - **I4** (frames): Treat *actually* and similar markers as cues that a statement contradicts an earlier one or an expected belief, and link them in prag to the frame they correct. - **I5** (learn): When humans or LLM teachers label targeted phenomena, mix in decoy items and check for annotator drift before accepting the silver data. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [University of British Columbia](https://md.research.elamur.ai/free/en/universities/university-of-british-columbia/) — University, Vancouver, Canada · 98 works · 34,805 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works 🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,263 works · [🧵 Discourse analysis](https://md.research.elamur.ai/free/en/topics/discourse-analysis/) · 466 works ## Cite Cite as: Emily Sadlier-Brown, Millie Lou, Miikka Silfverberg and Carla Kam. 2024. How Useful is Context, Actually? Comparing LLMs and Humans on Discourse Marker Prediction. Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{sadlierbrown2024how, title = {{How Useful is Context, Actually? Comparing LLMs and Humans on Discourse Marker Prediction}}, author = {Emily Sadlier-Brown, Millie Lou, Miikka Silfverberg and Carla Kam}, booktitle = {Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics}, year = {2024}, doi = {10.18653/v1/2024.cmcl-1.20}, url = {https://doi.org/10.18653/v1/2024.cmcl-1.20} } ``` - [← previous HeSum: a Novel Dataset for Abstractive Text Summarization in Hebrew](https://md.research.elamur.ai/free/en/papers/argaman-2024-hesum-novel-dataset-abstractive-text/) - [next → Logging Keystrokes in Writing by English Learners](https://md.research.elamur.ai/free/en/papers/velentzas-2024-logging-keystrokes-writing-english-learners/) - [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/10/) ## References 5 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Figure 3: The proportion of correctly identified actually answers as a function the number of examples completed. The regression line shows that the proportion of actually guesses does tend to increase as the number of completed examples increases.* Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774. [↗](https://arxiv.org/abs/2303.08774) 2. Karin Aijmer. 2002. English discourse particles. English Discourse Particles, pages 1–315. [↗](https://doi.org/10.1075/scl.10) 3. Douglas Biber and Edward Finegan. 1988. Adverbial stance types in english. Discourse Processes, 11(1):1–34. [↗](https://doi.org/10.1080/01638538809544689) 4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901. 5. H. Clark. 1996. Using language. Cambridge. *Figure 4: The distribution of the number of completed examples among participants. 58% of participants completed maximally five examples and 87% completed maximally fifteen.* [find](https://scholar.google.com/scholar?q=H%2E+Clark%2E+1996%2E+Using+language%2E+Cambridge%2E+%2AFigure+4%3A+The+distribution+of+the+number+of+completed+examples+among+participants%2E+58%25+of+participants+completed+maximally+five+examples+and+87%25+completed+m) *Updated 1 November 2026*