---
title: "Shared Logistic Normal Distributions for Soft Parameter Tying in"
description: "Unsupervised induction of dependency grammars usually puts Dirichlet priors on grammar weights. 30 references, cited by 61 papers here."
authors: ["Shay B. Cohen", "Noah A. Smith"]
year: 2009
venue: "Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics"
source: "https://aclanthology.org/N09-1009/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/cohen-2009-shared-logistic-normal-distributions-soft/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/cohen-2009-shared-logistic-normal-distributions-soft/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2009](https://md.research.elamur.ai/free/en/papers/2009/) / Shared Logistic Normal Distributions for Soft Parameter Tying in Unsupervised Grammar Induction

# Shared Logistic Normal Distributions for Soft Parameter Tying in Unsupervised Grammar Induction

**Authors:** [Shay B. Cohen](https://md.research.elamur.ai/free/en/authors/shay-b-cohen/) and [Noah A. Smith](https://md.research.elamur.ai/free/en/authors/noah-a-smith/) · **Year:** 2009 · **Venue:** [Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2009](https://md.research.elamur.ai/free/en/venues/naacl/2009/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/N09-1009/) · [PDF](https://aclanthology.org/N09-1009.pdf) · [OpenAlex](https://openalex.org/W2081228205)

**Citations:** 116 citations (OpenAlex, 2026-10-14) · [61](https://md.research.elamur.ai/free/en/papers/cohen-2009-shared-logistic-normal-distributions-soft/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Unsupervised induction of dependency grammars usually puts Dirichlet priors on grammar weights. Dirichlets cannot express beliefs about which rule probabilities should rise or fall together. An earlier logistic normal (LN) prior (Cohen et al., 2008) allowed such correlations, but only inside a single multinomial. Cohen and Smith generalize this into the shared logistic normal (SLN) distribution. In the SLN, several Gaussian "experts" are sampled and their components are averaged before the softmax, so weights in *different* multinomials can be softly tied. They derive a variational EM algorithm within an empirical Bayes framework and apply it to the DMV model of Klein and Manning (2004). In experiments on the Penn Treebank and the Chinese Treebank, tying distributions for heads of the same coarse category (verbs, nouns, adjectives) beats the LN, Dirichlet and EM baselines. They also train English and Chinese grammars jointly from non-parallel treebanks by sharing experts across languages, which gives the best accuracy for both. This shows that cross-lingual syntactic knowledge can be exploited without parallel text.

## Key points
- The model is the unlexicalized DMV for projective, unlabeled dependencies. It is trained fully unsupervised on POS-tag sequences, and evaluation uses attachment accuracy on sentences of length ≤10, ≤20 and all lengths.
- On English (Penn Treebank, all lengths), accuracy rises from 40.5% (LN baseline) to 41.4% with monolingual SLN tying of verbs and nouns, and to 42.2% with bilingual tying. The supervised MLE ceiling is 68.8%.
- For Chinese (Chinese Treebank), the best results also come from the bilingual setting, with the TIE A and TIE V&N variants.
- Tag sets are mapped onto twelve coarse families (noun, verb, adjective, etc.). Covariance matrices start at 1 on the diagonal, 0.5 for tags in the same family, and 0 elsewhere.
- Baselines are attach-right, EM (Klein & Manning, 2004), and Dirichlet and LN empirical Bayes (Cohen et al., 2008).

## Interesting ideas
- **Soft tying via shared Gaussian experts.** Several normal vectors are drawn and their entries are averaged before normalizing. Any parameters across separate distributions can then share statistical strength without being forced to be identical. This could serve any model built from many multinomials, such as HMMs, PCFGs or topic models.
- **Linguistic grouping as the prior's structure.** Coarse POS categories decide which distributions share an expert. This is a cheap way to inject typological or category knowledge into unsupervised learning, and it could suit low-resource languages like Ukrainian with UD-style universal tags.
- **Bilingual learning without parallel data.** Two grammars are merged into one model and tied only through a shared coarse tagset. Each language benefits from the other's patterns, which is attractive when parallel corpora are scarce.
- **Staged inference.** Inference first runs without tying and switches tying on only later. This saves computation and gives the expensive coupled model a sensible starting point, a trick that works for many non-concave objectives.
- **Informed covariance initialization.** Covariances begin with positive values within tag families instead of the identity matrix. This simple initialization helped, and it could be reused wherever correlated priors are learned.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 365 works · 158,048 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

## Cite

Cite as: Shay B. Cohen and Noah A. Smith. 2009. Shared Logistic Normal Distributions for Soft Parameter Tying in Unsupervised Grammar Induction. Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{cohen2009shared,
  title = {{Shared Logistic Normal Distributions for Soft Parameter Tying in Unsupervised Grammar Induction}},
  author = {Shay B. Cohen and Noah A. Smith},
  booktitle = {Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2009},
  doi = {10.3115/1620754.1620766},
  url = {https://doi.org/10.3115/1620754.1620766}
}
```

- [← previous Using a Dependency Parser to Improve SMT for Subject-Object-Verb Languages](https://md.research.elamur.ai/free/en/papers/xu-2009-dependency-parser-improve-smt-subject/)
- [next → Learning the Scope of Hedge Cues in Biomedical Texts](https://md.research.elamur.ai/free/en/papers/morante-2009-learning-scope-hedge-cues-biomedical/)
- [↑ back to papers of 2009](https://md.research.elamur.ai/free/en/papers/2009/)

## References

30 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. J. Aitchison. 1986. The Statistical Analysis of Compositional Data. Chapman and Hall, London. [find](https://scholar.google.com/scholar?q=J%2E+Aitchison%2E+1986%2E+The+Statistical+Analysis+of+Compositional+Data%2E+Chapman+and+Hall%2C+London%2E)
2. D. M. Blei and J. D. Lafferty. 2006. Correlated topic models. In Proc. of NIPS. [find](https://scholar.google.com/scholar?q=D%2E+M%2E+Blei+and+J%2E+D%2E+Lafferty%2E+2006%2E+Correlated+topic+models%2E+In+Proc%2E+of+NIPS%2E)
3. D. M. Blei, A. Ng, and M. Jordan. 2003. [Latent Dirichlet allocation](https://md.research.elamur.ai/free/en/papers/blei-2003-latent-dirichlet-allocation/). Journal of Machine Learning Research, 3:993–1022.
4. D. Burkett and D. Klein. 2008. Two languages are better than one (for syntactic parsing). In Proc. of EMNLP. [find](https://scholar.google.com/scholar?q=D%2E+Burkett+and+D%2E+Klein%2E+2008%2E+Two+languages+are+better+than+one+%28for+syntactic+parsing%29%2E+In+Proc%2E+of+EMNLP%2E)
5. E. Charniak and M. Johnson. 2005. Coarse-to-fine nbest parsing and maxent discriminative reranking. In Proc. of ACL. [find](https://scholar.google.com/scholar?q=E%2E+Charniak+and+M%2E+Johnson%2E+2005%2E+Coarse%2Dto%2Dfine+nbest+parsing+and+maxent+discriminative+reranking%2E+In+Proc%2E+of+ACL%2E)
6. S. B. Cohen and N. A. Smith. 2009. Inference for probabilistic grammars with shared logistic normal distributions. Technical report, Carnegie Mellon University. [find](https://scholar.google.com/scholar?q=S%2E+B%2E+Cohen+and+N%2E+A%2E+Smith%2E+2009%2E+Inference+for+probabilistic+grammars+with+shared+logistic+normal+distributions%2E+Technical+report%2C+Carnegie+Mellon+University%2E)
7. S. B. Cohen, K. Gimpel, and N. A. Smith. 2008. Logistic normal priors for unsupervised probabilistic grammar induction. In NIPS. [find](https://scholar.google.com/scholar?q=S%2E+B%2E+Cohen%2C+K%2E+Gimpel%2C+and+N%2E+A%2E+Smith%2E+2008%2E+Logistic+normal+priors+for+unsupervised+probabilistic+grammar+induction%2E+In+NIPS%2E)
8. M. Collins. 2003. Head-driven statistical models for natural language processing. Computational Linguistics, 29:589–637. [find](https://scholar.google.com/scholar?q=M%2E+Collins%2E+2003%2E+Head%2Ddriven+statistical+models+for+natural+language+processing%2E+Computational+Linguistics%2C+29%3A589%E2%80%93637%2E)
9. I. Dagan. 1991. [Two languages are more informative than one](https://md.research.elamur.ai/free/en/papers/dagan-1991-two-languages-are-more-informative/). In Proc. of ACL.
10. J. Eisner. 2002. Transformational priors over grammars. In Proc. of EMNLP. [find](https://scholar.google.com/scholar?q=J%2E+Eisner%2E+2002%2E+Transformational+priors+over+grammars%2E+In+Proc%2E+of+EMNLP%2E)
11. J. R. Finkel, T. Grenager, and C. D. Manning. 2007. The infinite tree. In Proc. of ACL. [find](https://scholar.google.com/scholar?q=J%2E+R%2E+Finkel%2C+T%2E+Grenager%2C+and+C%2E+D%2E+Manning%2E+2007%2E+The+infinite+tree%2E+In+Proc%2E+of+ACL%2E)
12. J. Goodman. 1996. Parsing algorithms and metrics. In Proc. of ACL. [find](https://scholar.google.com/scholar?q=J%2E+Goodman%2E+1996%2E+Parsing+algorithms+and+metrics%2E+In+Proc%2E+of+ACL%2E)
13. A. Haghighi, P. Liang, T. Berg-Kirkpatrick, and D. Klein. 2008. [Learning bilingual lexicons from monolingual corpora](https://md.research.elamur.ai/free/en/papers/haghighi-2008-learning-bilingual-lexicons-monolingual-corpora/). In Proc. of ACL.
14. W. P. Headden, M. Johnson, and D. McClosky. 2009. Improving unsupervised dependency parsing with richer contexts and smoothing. In Proc. of NAACL-HLT. [find](https://scholar.google.com/scholar?q=W%2E+P%2E+Headden%2C+M%2E+Johnson%2C+and+D%2E+McClosky%2E+2009%2E+Improving+unsupervised+dependency+parsing+with+richer+contexts+and+smoothing%2E+In+Proc%2E+of+NAACL%2DHLT%2E)
15. G. E. Hinton. 1999. Products of experts. In Proc. of ICANN. [find](https://scholar.google.com/scholar?q=G%2E+E%2E+Hinton%2E+1999%2E+Products+of+experts%2E+In+Proc%2E+of+ICANN%2E)
16. M. Johnson, T. L. Griffiths, and S. Goldwater. 2006. Adaptor grammars: A framework for specifying compositional nonparameteric Bayesian models. In NIPS. [find](https://scholar.google.com/scholar?q=M%2E+Johnson%2C+T%2E+L%2E+Griffiths%2C+and+S%2E+Goldwater%2E+2006%2E+Adaptor+grammars%3A+A+framework+for+specifying+compositional+nonparameteric+Bayesian+models%2E+In+NIPS%2E)
17. M. Johnson, T. L. Griffiths, and S. Goldwater. 2007. [Bayesian inference for PCFGs via Markov chain Monte Carlo](https://md.research.elamur.ai/free/en/papers/johnson-2007-bayesian-inference-pcfgs-markov-chain/). In Proc. of NAACL.
18. M. Johnson. 2007. [Why doesn’t EM find good HMM POS-taggers?](https://md.research.elamur.ai/free/en/papers/johnson-2007-why-doesn-t-em-find/) In Proc. EMNLP-CoNLL.
19. M. I. Jordan, Z. Ghahramani, T. S. Jaakola, and L. K. Saul. 1999. An introduction to variational methods for graphical models. Machine Learning, 37(2):183– 233. [find](https://scholar.google.com/scholar?q=M%2E+I%2E+Jordan%2C+Z%2E+Ghahramani%2C+T%2E+S%2E+Jaakola%2C+and+L%2E+K%2E+Saul%2E+1999%2E+An+introduction+to+variational+methods+for+graphical+models%2E+Machine+Learning%2C+37%282%29%3A183%E2%80%93+233%2E)
20. D. Klein and C. D. Manning. 2004. Corpus-based induction of syntactic structure: Models of dependency and constituency. In Proc. of ACL. [find](https://scholar.google.com/scholar?q=D%2E+Klein+and+C%2E+D%2E+Manning%2E+2004%2E+Corpus%2Dbased+induction+of+syntactic+structure%3A+Models+of+dependency+and+constituency%2E+In+Proc%2E+of+ACL%2E)
21. K. Kurihara and T. Sato. 2006. Variational Bayesian grammar induction for natural language. In Proc. of ICGI. [find](https://scholar.google.com/scholar?q=K%2E+Kurihara+and+T%2E+Sato%2E+2006%2E+Variational+Bayesian+grammar+induction+for+natural+language%2E+In+Proc%2E+of+ICGI%2E)
22. P. Liang, S. Petrov, M. Jordan, and D. Klein. 2007. The infinite PCFG using hierarchical Dirichlet processes. In Proc. of EMNLP. [find](https://scholar.google.com/scholar?q=P%2E+Liang%2C+S%2E+Petrov%2C+M%2E+Jordan%2C+and+D%2E+Klein%2E+2007%2E+The+infinite+PCFG+using+hierarchical+Dirichlet+processes%2E+In+Proc%2E+of+EMNLP%2E)
23. M. P. Marcus, B. Santorini, and M. A. Marcinkiewicz. 1993. [Building a large annotated corpus of English: The Penn treebank](https://md.research.elamur.ai/free/en/papers/marcus-1993-building-large-annotated-corpus-english/). Computational Linguistics, 19:313–330.
24. D. A. Smith and N. A. Smith. 2004. Bilingual parsing with factored estimation: Using English to parse Korean. In Proc. of EMNLP, pages 49–56. [find](https://scholar.google.com/scholar?q=D%2E+A%2E+Smith+and+N%2E+A%2E+Smith%2E+2004%2E+Bilingual+parsing+with+factored+estimation%3A+Using+English+to+parse+Korean%2E+In+Proc%2E+of+EMNLP%2C+pages+49%E2%80%9356%2E)
25. N. A. Smith. 2006. Novel Estimation Methods for Unsupervised Discovery of Latent Structure in Natural Language Text. Ph.D. thesis, Johns Hopkins University. [find](https://scholar.google.com/scholar?q=N%2E+A%2E+Smith%2E+2006%2E+Novel+Estimation+Methods+for+Unsupervised+Discovery+of+Latent+Structure+in+Natural+Language+Text%2E+Ph%2ED%2E+thesis%2C+Johns+Hopkins+University%2E)
26. B. Snyder and R. Barzilay. 2008. [Unsupervised multilingual learning for morphological segmentation](https://md.research.elamur.ai/free/en/papers/snyder-2008-unsupervised-multilingual-learning-morphological-segmentation/). In Proc. of ACL.
27. Y. W. Teh. 2006. [A hierarchical Bayesian language model based on Pitman-Yor processes](https://md.research.elamur.ai/free/en/papers/teh-2006-hierarchical-bayesian-language-model-based/). In Proc. of COLING-ACL.
28. M. Wang, N. A. Smith, and T. Mitamura. 2007. What is the Jeopardy model? a quasi-synchronous grammar for question answering. In Proc. of EMNLP. [find](https://scholar.google.com/scholar?q=M%2E+Wang%2C+N%2E+A%2E+Smith%2C+and+T%2E+Mitamura%2E+2007%2E+What+is+the+Jeopardy+model%3F+a+quasi%2Dsynchronous+grammar+for+question+answering%2E+In+Proc%2E+of+EMNLP%2E)
29. D. Wu. 1997. [Stochastic inversion transduction grammars and bilingual parsing of parallel corpora](https://md.research.elamur.ai/free/en/papers/wu-1997-stochastic-inversion-transduction-grammars-bilingual/). Comp. Ling., 23(3):377–404.
30. N. Xue, F. Xia, F.-D. Chiou, and M. Palmer. 2004. The Penn Chinese Treebank: Phrase structure annotation of a large corpus. Natural Language Engineering, 10(4):1–30. [find](https://scholar.google.com/scholar?q=N%2E+Xue%2C+F%2E+Xia%2C+F%2E%2DD%2E+Chiou%2C+and+M%2E+Palmer%2E+2004%2E+The+Penn+Chinese+Treebank%3A+Phrase+structure+annotation+of+a+large+corpus%2E+Natural+Language+Engineering%2C+10%284%29%3A1%E2%80%9330%2E)

*Updated 24 October 2026*