---
title: "Unlocking Context Constraints of LLMs — paper by Li (2023)"
description: "The paper proposes Selective Context , which shrinks the input before it reaches the LLM. 0.3 BLEU. 8 references, cited by 23 papers here."
authors: ["Yucheng Li"]
year: 2023
venue: "arXiv:2304.12102"
source: "https://arxiv.org/abs/2304.12102"
doi: "10.48550/arxiv.2304.12102"
arxiv: "2304.12102"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/li-2023-unlocking-context-constraints-llms-enhancing/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/li-2023-unlocking-context-constraints-llms-enhancing/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/9/) / Unlocking Context Constraints of LLMs: Enhancing Context Efficiency of LLMs with Self-Information-Based Content Filtering

# Unlocking Context Constraints of LLMs: Enhancing Context Efficiency of LLMs with Self-Information-Based Content Filtering

**Authors:** [Yucheng Li](https://md.research.elamur.ai/free/en/authors/yucheng-li/) · **Year:** 2023 · **Venue:** arXiv:2304.12102 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2304.12102) · [PDF](https://arxiv.org/pdf/2304.12102) · [DOI 10.48550/arxiv.2304.12102](https://doi.org/10.48550/arxiv.2304.12102) · [arXiv 2304.12102](https://arxiv.org/abs/2304.12102) · [OpenAlex](https://openalex.org/W4367000321)

**Citations:** 9 citations (OpenAlex, 2026-10-14) · [23](https://md.research.elamur.ai/free/en/papers/li-2023-unlocking-context-constraints-llms-enhancing/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Transformer-based LLMs can only attend to a fixed window of tokens, and enlarging that window is costly because attention scales quadratically. This makes long documents and lengthy dialogues hard to handle. The paper proposes **Selective Context**, which shrinks the input before it reaches the LLM. A smaller causal language model (e.g., GPT-2, OPT, LLaMA) scores every token by self-information, its negative log-probability given the preceding text. Because self-information is additive, these scores are summed into larger units: tokens, noun phrases (merged with spaCy) or sentences. Units falling below a chosen percentile of the score distribution are dropped. The method was tested on recent arXiv papers, BBC News articles and ShareGPT conversations, all collected after ChatGPT's knowledge cutoff to limit contamination. Tasks were reconstruction, summarisation, QA and conversation, scored with BLEU, ROUGE-1 and BERTScore. Removing roughly 20–35% of the content causes only small quality losses and beats random deletion. This offers a cheap, model-agnostic way to stretch limited context budgets.

## Key points
- **Datasets:** Arxiv has 408 documents (only the first two sections are used), BBC News has 294, and ShareGPT has 470 conversations. Each document averages about 690–865 tokens.
- **Example compression:** Filtering at the phrase level with the percentile set to 50 kept about 57% of the tokens, saving roughly 43% of the context.
- **Small losses at 20% reduction:** Compared with answers from the full context, BLEU fell by only 0.05 and ROUGE-1 by 0.03. Selective Context also outperformed a random-deletion baseline at 35% reduction (about 0.3 BLEU and over 0.55 ROUGE-1).
- **Task sensitivity:** Reconstruction reached ROUGE-1 of about 0.65 and BERTScore above 0.9 at reduction ratios of 0.2–0.35, but collapsed at 0.8 (BLEU 0.03). Summarisation and conversation held up well, while QA and reconstruction were more sensitive because they need fine details.
- **Code:** Available at github.com/liyucheng09/Selective_Context.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Yucheng Li. 2023. Unlocking Context Constraints of LLMs: Enhancing Context Efficiency of LLMs with Self-Information-Based Content Filtering. arXiv:2304.12102.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{li2023unlocking,
  title = {{Unlocking Context Constraints of LLMs: Enhancing Context Efficiency of LLMs with Self-Information-Based Content Filtering}},
  author = {Yucheng Li},
  year = {2023},
  doi = {10.48550/arxiv.2304.12102},
  eprint = {2304.12102},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2304.12102}
}
```

- [← previous Seq2seq is All You Need for Coreference Resolution](https://md.research.elamur.ai/free/en/papers/zhang-2023-seq2seq-is-all-you-need/)
- [next → Annotation-Inspired Implicit Discourse Relation Classification with Auxiliary Discourse Connective Generation](https://md.research.elamur.ai/free/en/papers/liu-2023-annotation-inspired-implicit-discourse-relation/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/9/)

## References

8 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Banerjee, S. and Lavie, A. [Meteor: An automatic metric for mt evaluation with improved correlation with human judgments](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/). In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65–72, 2005.
2. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33: 1877–1901, 2020.
3. Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. [Sparks of artificial general intelligence: Early experiments with gpt-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/). arXiv preprint arXiv:2303.12712, 2023. \#\#\# 5.4. Case Study Dong, Z., Tang, T., Li, L., and Zhao, W…
4. To have a more straightforward impression of how Selective Context reduce context cost, we present several cases in the Appendix. Lin, C.-Y. [Rouge: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text summarization branches out, pp. 74–81, 2004. \#\# 6. Conclusion Papineni, K., Roukos, S., Ward, T…
5. In this paper, we introduced Selective Context to maximise the utility of fixed context length in LLMs. We demonstrated the effectiveness of our method by filtering out less informative content, providing a more compact and efficient context representation for LLMs without sacrificing their Shannon,… [find](https://scholar.google.com/scholar?q=In+this+paper%2C+we+introduced+Selective+Context+to+maximise+the+utility+of+fixed+context+length+in+LLMs%2E+We+demonstrated+the+effectiveness+of+our+method+by+filtering+out+less+informative+content%2C+provi)
6. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023.
7. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30, 2017.
8. Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019. *Figure 6. Selective Context with reduction ratio of 0.5 on conversation transcripts.* [find](https://scholar.google.com/scholar?q=Zhang%2C+T%2E%2C+Kishore%2C+V%2E%2C+Wu%2C+F%2E%2C+Weinberger%2C+K%2E+Q%2E%2C+and+Artzi%2C+Y%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+arXiv+preprint+arXiv%3A1904%2E09675%2C+2019%2E+%2AFigure+6%2E+Selective+Context+with+reduction+rat)

*Updated 30 October 2026*