---
title: "Dynamic Language Models for Streaming Text — Yogatama 2014"
description: "Yogatama et al. propose a probabilistic language model whose parameters evolve daily and can… 26 references, cited by 10 papers here."
authors: ["Dani Yogatama", "Chong Wang", "Bryan R. Routledge", "Noah A. Smith", "Eric P. Xing"]
year: 2014
venue: "Transactions of the Association for Computational Linguistics"
source: "https://aclanthology.org/Q14-1015/"
doi: "10.1162/tacl_a_00175"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yogatama-2014-dynamic-language-models-streaming-text/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yogatama-2014-dynamic-language-models-streaming-text/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2014](https://md.research.elamur.ai/free/en/papers/2014/page/3/) / Dynamic Language Models for Streaming Text

# Dynamic Language Models for Streaming Text

**Authors:** [Dani Yogatama](https://md.research.elamur.ai/free/en/authors/dani-yogatama/), [Chong Wang](https://md.research.elamur.ai/free/en/authors/chong-wang/), [Bryan R. Routledge](https://md.research.elamur.ai/free/en/authors/bryan-routledge/), [Noah A. Smith](https://md.research.elamur.ai/free/en/authors/noah-a-smith/) and [Eric P. Xing](https://md.research.elamur.ai/free/en/authors/eric-p-xing/) · **Year:** 2014 · **Venue:** [Transactions of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/tacl/) ([Transactions of the Association for Computational Linguistics (TACL) 2014](https://md.research.elamur.ai/free/en/venues/tacl/2014/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/Q14-1015/) · [PDF](https://aclanthology.org/Q14-1015.pdf) · [DOI 10.1162/tacl\_a\_00175](https://doi.org/10.1162/tacl_a_00175) · [OpenAlex](https://openalex.org/W2151835407)

**Citations:** 22 citations (OpenAlex, 2026-10-14) · 10 in our library, of the 149,439 papers whose bibliographies we read

## Summary
Most language models assume word distributions stay fixed over time. That assumption fits poorly with fast-changing streams like news and social media. Yogatama et al. propose a probabilistic language model whose parameters evolve daily and can depend on observable non-text signals, here stock market movements. Each day's word distribution combines background log-frequencies with a deviation vector. That vector is a weighted mixture of deviations from a fixed window of recent days. The weights reflect how closely current market conditions resemble those earlier days, scored by learned feature coefficients. Training is fully online. It combines mean-field variational inference with ideas from regret-minimizing sequential prediction, so old data need not be stored. The authors define a new task: predicting a day's economics-related text given stock prices up to that day. They test unigram and bigram base models on five streams: Reuters economic news in English, French and German, plus Twitter. Their model beats competing baselines in perplexity across these streams. The work offers a scalable way to link text modeling with numeric real-world context.

## Key points
- Data: four Reuters news streams (EN:NA, EN:JP, FR, DE) and one Twitter stream, with stock price changes as context features.
- The default history window is c = 7 days. Expanding it to c = 14 brought no gain, and the model selects the t − 1 day's distribution most often.
- On Twitter, deviation coefficients for terms like goog, #goog, @google and #microsoft rise around sharp moves in Google and Microsoft stock returns. Highly weighted stocks include ConAgra Foods, Intel and Amazon.com.
- The approach works with any base model. Bigrams are handled with one β per history word, and the method relates to dynamic topic models and the online model of Iwata et al. (2010).
- Single-membership topic variables did not improve perplexity and raised computational cost. LDA-style extensions are left for future work.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 539 works · 179,533 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Dani Yogatama, Chong Wang, Bryan R. Routledge, Noah A. Smith and Eric P. Xing. 2014. Dynamic Language Models for Streaming Text. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{yogatama2014dynamic,
  title = {{Dynamic Language Models for Streaming Text}},
  author = {Dani Yogatama, Chong Wang, Bryan R. Routledge, Noah A. Smith and Eric P. Xing},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2014},
  doi = {10.1162/tacl\_a\_00175},
  url = {https://doi.org/10.1162/tacl_a_00175}
}
```

- [← previous Advances in dialectal Arabic speech recognition: a study using Twitter to improve Egyptian ASR](https://md.research.elamur.ai/free/en/papers/ali-2014-advances-dialectal-arabic-speech-recognition/)
- [next → Reducing Sparsity Improves the Recognition of Implicit Discourse Relations](https://md.research.elamur.ai/free/en/papers/li-2014-reducing-sparsity-improves-recognition-implicit/)
- [↑ back to papers of 2014](https://md.research.elamur.ai/free/en/papers/2014/page/3/)

## References

26 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Galen Andrew and Jianfeng Gao. 2007. Scalable training of l 1 -regularized log-linear models. In Proc. of ICML. [find](https://scholar.google.com/scholar?q=Galen+Andrew+and+Jianfeng+Gao%2E+2007%2E+Scalable+training+of+l+1+%2Dregularized+log%2Dlinear+models%2E+In+Proc%2E+of+ICML%2E)
2. David M. Blei and John D. Lafferty. 2006. [Dynamic topic models](https://md.research.elamur.ai/free/en/papers/blei-2006-dynamic-topic-models/). In Proc. of ICML.
3. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. [Latent Dirichlet allocation](https://md.research.elamur.ai/free/en/papers/blei-2003-latent-dirichlet-allocation/). Journal of Machine Learning Research, 3:993–1022. Sébastien Bubeck. 2011. Introduction to online optimization. Technical report, Department of Operations Research and Financial Engineering, Princeton University.…
4. Prediction, Learning, and Games. Cambridge University Press. William S. Cleveland. 1979. Robust locally weighted regression and smoothing scatterplots. Journal of the American Statistical Association, 74(368):829–836. [find](https://scholar.google.com/scholar?q=Prediction%2C+Learning%2C+and+Games%2E+Cambridge+University+Press%2E+William+S%2E+Cleveland%2E+1979%2E+Robust+locally+weighted+regression+and+smoothing+scatterplots%2E+Journal+of+the+American+Statistical+Association%2C)
5. Thomas M. Cover and Joy A. Thomas. 1991. Elements of Information Theory. [find](https://scholar.google.com/scholar?q=Thomas+M%2E+Cover+and+Joy+A%2E+Thomas%2E+1991%2E+Elements+of+Information+Theory%2E)
6. John Wiley & Sons. John Duchi and Yoram Singer. 2009. Efficient online and batch learning using forward backward splitting. Journal of Machine Learning Research, 10(7):2899– 2934. [find](https://scholar.google.com/scholar?q=John+Wiley+%26+Sons%2E+John+Duchi+and+Yoram+Singer%2E+2009%2E+Efficient+online+and+batch+learning+using+forward+backward+splitting%2E+Journal+of+Machine+Learning+Research%2C+10%287%29%3A2899%E2%80%93+2934%2E)
7. Jacob Eisenstein, Brendan O’Connor, Noah A. Smith, and Eric P. Xing. 2010. [A latent variable model for geographic lexical variation](https://md.research.elamur.ai/free/en/papers/eisenstein-2010-latent-variable-model-geographic-lexical/). In Proc. of EMNLP.
8. Jacob Eisenstein, Amr Ahmed, and Eric P. Xing. 2011. Sparse additive generative models of text. In Proc. of ICML. [find](https://scholar.google.com/scholar?q=Jacob+Eisenstein%2C+Amr+Ahmed%2C+and+Eric+P%2E+Xing%2E+2011%2E+Sparse+additive+generative+models+of+text%2E+In+Proc%2E+of+ICML%2E)
9. Amit Goyal, Hal Daume III, and Suresh Venkatasubramanian. 2009. Streaming for large scale NLP: Language modeling. In Proc. of HLT-NAACL. [find](https://scholar.google.com/scholar?q=Amit+Goyal%2C+Hal+Daume+III%2C+and+Suresh+Venkatasubramanian%2E+2009%2E+Streaming+for+large+scale+NLP%3A+Language+modeling%2E+In+Proc%2E+of+HLT%2DNAACL%2E)
10. Matt Hoffman, David M. Blei, Chong Wang, and John Paisley. 2013. Stochastic variational inference. Journal of Machine Learning Research, 14:1303–1347. [find](https://scholar.google.com/scholar?q=Matt+Hoffman%2C+David+M%2E+Blei%2C+Chong+Wang%2C+and+John+Paisley%2E+2013%2E+Stochastic+variational+inference%2E+Journal+of+Machine+Learning+Research%2C+14%3A1303%E2%80%931347%2E)
11. Antti Honkela and Harri Valpola. 2003. On-line variational Bayesian learning. In Proc. of ICA. [find](https://scholar.google.com/scholar?q=Antti+Honkela+and+Harri+Valpola%2E+2003%2E+On%2Dline+variational+Bayesian+learning%2E+In+Proc%2E+of+ICA%2E)
12. Tomoharu Iwata, Takeshi Yamada, Yasushi Sakurai, and Naonori Ueda. 2010. Online multiscale dynamic topic models. In Proc. of KDD. [find](https://scholar.google.com/scholar?q=Tomoharu+Iwata%2C+Takeshi+Yamada%2C+Yasushi+Sakurai%2C+and+Naonori+Ueda%2E+2010%2E+Online+multiscale+dynamic+topic+models%2E+In+Proc%2E+of+KDD%2E)
13. Frederick Jelinek. 1997. Statistical Methods for Speech Recognition. [find](https://scholar.google.com/scholar?q=Frederick+Jelinek%2E+1997%2E+Statistical+Methods+for+Speech+Recognition%2E)
14. MIT Press. Jyrki Kivinen and Manfred K. Warmuth. 1997. Exponentiated gradient versus gradient descent for linear predictors. Information and Computation, 132:1–63. [find](https://scholar.google.com/scholar?q=MIT+Press%2E+Jyrki+Kivinen+and+Manfred+K%2E+Warmuth%2E+1997%2E+Exponentiated+gradient+versus+gradient+descent+for+linear+predictors%2E+Information+and+Computation%2C+132%3A1%E2%80%9363%2E)
15. Victor Lavrenko, Matt Schmill, Dawn Lawrie, Paul Ogilvie, David Jensen, and James Allan. 2000. Mining of concurrent text and time series. In Proc. of KDD Workshop on Text Mining. [find](https://scholar.google.com/scholar?q=Victor+Lavrenko%2C+Matt+Schmill%2C+Dawn+Lawrie%2C+Paul+Ogilvie%2C+David+Jensen%2C+and+James+Allan%2E+2000%2E+Mining+of+concurrent+text+and+time+series%2E+In+Proc%2E+of+KDD+Workshop+on+Text+Mining%2E)
16. Abby Levenberg and Miles Osborne. 2009. Stream-based randomised language models for SMT. In Proc. of EMNLP. [find](https://scholar.google.com/scholar?q=Abby+Levenberg+and+Miles+Osborne%2E+2009%2E+Stream%2Dbased+randomised+language+models+for+SMT%2E+In+Proc%2E+of+EMNLP%2E)
17. Abby Levenberg, Chris Callison-Burch, and Miles Osborne. 2010. Stream-based translation models for statistical machine translation. In Proc. of HLT-NAACL. [find](https://scholar.google.com/scholar?q=Abby+Levenberg%2C+Chris+Callison%2DBurch%2C+and+Miles+Osborne%2E+2010%2E+Stream%2Dbased+translation+models+for+statistical+machine+translation%2E+In+Proc%2E+of+HLT%2DNAACL%2E)
18. Dong C. Liu and Jorge Nocedal. 1989. On the limited memory BFGS method for large scale optimization. Mathematical Programming B, 45(3):503–528. [find](https://scholar.google.com/scholar?q=Dong+C%2E+Liu+and+Jorge+Nocedal%2E+1989%2E+On+the+limited+memory+BFGS+method+for+large+scale+optimization%2E+Mathematical+Programming+B%2C+45%283%29%3A503%E2%80%93528%2E)
19. David Mimno and Andrew McCallum. 2008. Topic models conditioned on arbitrary features with Dirichlet-multinomial regression. In Proc. of UAI. [find](https://scholar.google.com/scholar?q=David+Mimno+and+Andrew+McCallum%2E+2008%2E+Topic+models+conditioned+on+arbitrary+features+with+Dirichlet%2Dmultinomial+regression%2E+In+Proc%2E+of+UAI%2E)
20. Alexander Rakhlin. 2009. Lecture notes on online learning. Technical report, Department of Statistics, The Wharton School, University of Pennsylvania. [find](https://scholar.google.com/scholar?q=Alexander+Rakhlin%2E+2009%2E+Lecture+notes+on+online+learning%2E+Technical+report%2C+Department+of+Statistics%2C+The+Wharton+School%2C+University+of+Pennsylvania%2E)
21. Masaaki Sato. 2001. Online model selection based on the variational bayes. Neural Computation, 13(7):1649– 1681. [find](https://scholar.google.com/scholar?q=Masaaki+Sato%2E+2001%2E+Online+model+selection+based+on+the+variational+bayes%2E+Neural+Computation%2C+13%287%29%3A1649%E2%80%93+1681%2E)
22. Shai Shalev-Shwartz. 2012. Online learning and online convex optimization. Foundations and Trends in Machine Learning, 4(2):107–194. [find](https://scholar.google.com/scholar?q=Shai+Shalev%2DShwartz%2E+2012%2E+Online+learning+and+online+convex+optimization%2E+Foundations+and+Trends+in+Machine+Learning%2C+4%282%29%3A107%E2%80%93194%2E)
23. Martin J. Wainwright and Michael I. Jordan. 2008. Graphical models, exponential families, and variational inference. Foundations and Trends in Machine Learning, 1(1–2):1–305. [find](https://scholar.google.com/scholar?q=Martin+J%2E+Wainwright+and+Michael+I%2E+Jordan%2E+2008%2E+Graphical+models%2C+exponential+families%2C+and+variational+inference%2E+Foundations+and+Trends+in+Machine+Learning%2C+1%281%E2%80%932%29%3A1%E2%80%93305%2E)
24. Chong Wang, David M. Blei, and David Heckerman. 2008. Continuous time dynamic topic models. In Proc. of UAI. [find](https://scholar.google.com/scholar?q=Chong+Wang%2C+David+M%2E+Blei%2C+and+David+Heckerman%2E+2008%2E+Continuous+time+dynamic+topic+models%2E+In+Proc%2E+of+UAI%2E)
25. Chong Wang, John Paisley, and David M. Blei. 2011. Online variational inference for the hierarchical Dirichlet process. In Proc. of AISTATS. [find](https://scholar.google.com/scholar?q=Chong+Wang%2C+John+Paisley%2C+and+David+M%2E+Blei%2E+2011%2E+Online+variational+inference+for+the+hierarchical+Dirichlet+process%2E+In+Proc%2E+of+AISTATS%2E)
26. Martin Zinkevich. 2003. Online convex programming and generalized infinitesimal gradient ascent. In Proc. of ICML. [find](https://scholar.google.com/scholar?q=Martin+Zinkevich%2E+2003%2E+Online+convex+programming+and+generalized+infinitesimal+gradient+ascent%2E+In+Proc%2E+of+ICML%2E)

*Updated 30 October 2026*