---
title: "Domain Adaptation with Structural — Blitzer 2006"
description: "The paper addresses domain adaptation for discriminative NLP models: a model trained on a well-annotated source domain often performs poorly on a target…"
authors: ["John Blitzer", "Ryan McDonald", "Fernando Pereira"]
year: 2006
venue: "Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/W06-1615/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/blitzer-2006-domain-adaptation-structural-correspondence-learning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/blitzer-2006-domain-adaptation-structural-correspondence-learning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2006](https://md.research.elamur.ai/free/en/papers/2006/) / Domain Adaptation with Structural Correspondence Learning

# Domain Adaptation with Structural Correspondence Learning

**Authors:** [John Blitzer](https://md.research.elamur.ai/free/en/authors/john-blitzer/), [Ryan McDonald](https://md.research.elamur.ai/free/en/authors/ryan-mcdonald/), [Fernando Pereira](https://md.research.elamur.ai/free/en/authors/fernando-pereira/) · **Year:** 2006 · **Venue:** Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W06-1615/) · [PDF](https://aclanthology.org/W06-1615.pdf) · [OpenAlex](https://openalex.org/W2158108973)

**Citations:** 1527 citations (openalex-snapshot · 2026-10-14) · [265](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
The paper addresses domain adaptation for discriminative NLP models: a model trained on a well-annotated source domain often performs poorly on a target domain that has a different vocabulary and little or no labeled data. The authors propose structural correspondence learning (SCL). SCL uses unlabeled text from both domains to learn a shared low-dimensional feature representation. They first choose pivot features, which are frequent in both domains and behave alike in both. They then train linear classifiers that predict each pivot from the other features. An SVD of the stacked classifier weights gives a projection that links domain-specific features related to the same pivots. The projected features are added to the original ones when training a source-domain classifier. In the experiments, a part-of-speech tagger trained on Wall Street Journal text is adapted to MEDLINE abstracts. With no labeled target data, SCL beats both a supervised tagger and a semi-supervised ASO baseline. It also combines with target labeled data through the classifier combination of Florian et al. (2004), and the better tags raise parsing accuracy on biomedical text. The method matters because it needs no labeled data in the new domain.

## Key points
- The setting is adapting a PoS tagger from WSJ (financial news) to MEDLINE (biomedical abstracts). Test sets come from PennBioIE annotations of MEDLINE.
- Pivot features are left, right and middle tokens for words that are frequent in both domains, giving many thousands of pivots.
- Pivot predictors are trained with the modified Huber loss. Features that trivially reveal the pivot are excluded from its predictor.
- Implementation choices:
  - The projection dimension is h = 25; values from 20 to 100 changed little.
  - Only positive weights enter the SVD.
  - Block SVDs are computed separately for each of the 58 Ratnaparkhi (1996) feature types, giving 1450 projection features.
- SCL beats supervised and ASO taggers without target labels. Top-k ASO, a co-training-like variant, failed to beat the supervised baseline. The SCL tagger also improves downstream parsing on MEDLINE.

## Interesting ideas
- **Pivots as a bridge between vocabularies:** Context features that are frequent in both domains act as anchors. Unseen target words, such as biomedical nouns, can then be aligned with familiar source words that share those contexts. This could help adapt Ukrainian taggers from news to legal, medical or trade texts.
- **Self-supervised auxiliary tasks from unlabeled text:** Predicting whether a pivot is present needs no annotation. Masking the pivot's own feature forces the predictor to learn real correlations instead of trivial ones. The same trick suits any task where cheap proxy targets can be built from raw data.
- **Augment, don't replace:** The projected features are appended to the original ones. The representation can therefore be added to an existing classifier, or combined with other adaptation methods, without discarding what already works.
- **Rescaling learned features:** A single scaling factor, tuned on source held-out data, gives the dense projected features enough weight under regularization. The authors found this essential to reproduce the ASO results. It is a useful check whenever dense learned features are mixed with sparse binary ones.
- **Extrinsic evaluation through a pipeline:** The authors measure gains in tagging and also in downstream parsing. This shows whether an improvement in an upstream component actually carries through to the end task.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Part-of-speech tagging](https://md.research.elamur.ai/free/en/topics/part-of-speech-tagging/)

## Cited by

Cited by

265 papers of our library cite it (of the 140884 whose bibliographies we read); 20 of them have a page here, 245 not yet.

- [Extending a Parser to Distant Domains Using a Few Dozen Partially Annotated Examples](https://md.research.elamur.ai/free/en/papers/acl-2018-joshi-extending-parser-distant-domains-few/)
- [Part-of-Speech Tagging for Historical English](https://md.research.elamur.ai/free/en/papers/aclbulk-2016-yang-part-speech-tagging-historical-english/)
- [Biographies, Bollywood, Boom-boxes and Blenders: Domain Adaptation for Sentiment Classification](https://md.research.elamur.ai/free/en/papers/blitzer-2007-biographies-bollywood-boom-boxes-blenders/)
- [On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/)
- [Exploring Difficulties in Parsing Imperatives and Questions](https://md.research.elamur.ai/free/en/papers/hara-2011-exploring-difficulties-parsing-imperatives-questions/)
- [Cross-Domain Label-Adaptive Stance Detection](https://md.research.elamur.ai/free/en/papers/hardalov-2021-cross-domain-label-adaptive-stance/)
- [Out-of-domain FrameNet Semantic Role Labeling](https://md.research.elamur.ai/free/en/papers/hartmann-2017-out-domain-framenet-semantic-role/)
- [Demographic Factors Improve Classification Performance](https://md.research.elamur.ai/free/en/papers/hovy-2015-demographic-factors-improve-classification-performance/)
- [Frustratingly Easy Domain Adaptation](https://md.research.elamur.ai/free/en/papers/iii-2007-frustratingly-easy-domain-adaptation/)
- [Closing the Gap: Domain Adaptation from Explicit to Implicit Discourse Relations](https://md.research.elamur.ai/free/en/papers/ji-2015-closing-gap-domain-adaptation-explicit/)
- [Instance Weighting for Domain Adaptation in NLP](https://md.research.elamur.ai/free/en/papers/jiang-2007-instance-weighting-domain-adaptation-nlp/)
- [FlauBERT: Unsupervised Language Model Pre-training for French](https://md.research.elamur.ai/free/en/papers/le-2019-flaubert-unsupervised-language-model-pre/)
- [A Holistic Approach to Undesired Content Detection in the Real World](https://md.research.elamur.ai/free/en/papers/markov-2022-holistic-approach-undesired-content-detection/)
- [Event Detection and Domain Adaptation with Convolutional Neural Networks](https://md.research.elamur.ai/free/en/papers/nguyen-2015-event-detection-domain-adaptation-convolutional/)
- [Relation Extraction: Perspective from Convolutional Neural Networks](https://md.research.elamur.ai/free/en/papers/nguyen-2015-relation-extraction-perspective-convolutional-neural/)
- [The CoNLL 2007 Shared Task on Dependency Parsing](https://md.research.elamur.ai/free/en/papers/nivre-2007-conll-2007-shared-task-dependency/)
- [Neural Unsupervised Domain Adaptation in NLP–-A Survey](https://md.research.elamur.ai/free/en/papers/ramponi-2020-neural-unsupervised-domain-adaptation-nlp/)
- [Polyglot: Distributed Word Representations for Multilingual NLP](https://md.research.elamur.ai/free/en/papers/rfou-2013-polyglot-distributed-word-representations-multilingual/)
- [MASS: Masked Sequence to Sequence Pre-training for Language Generation](https://md.research.elamur.ai/free/en/papers/song-2019-mass-masked-sequence-sequence-pre/)
- [Domain Adaptation for Coreference Resolution: An Adaptive Ensemble Approach](https://md.research.elamur.ai/free/en/papers/yang-2012-domain-adaptation-coreference-resolution-adaptive/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{blitzer2006domain,
  title = {{Domain Adaptation with Structural Correspondence Learning}},
  author = {Blitzer, John and McDonald, Ryan and Pereira, Fernando},
  booktitle = {Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing},
  year = {2006},
  doi = {10.3115/1610075.1610094},
  url = {https://doi.org/10.3115/1610075.1610094}
}
```

- [← previous Discriminant-Based MRS Banking](https://md.research.elamur.ai/free/en/papers/lingbulk-2006-oepen-discriminant-based-mrs-banking/)
- [next → Dynamic Topic Models](https://md.research.elamur.ai/free/en/papers/blei-2006-dynamic-topic-models/)
- [↑ back to papers of 2006](https://md.research.elamur.ai/free/en/papers/2006/)

## References

25 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. R. Ando and T. Zhang. 2005a. A framework for learning predictive structures from multiple tasks and unlabeled data. JMLR, 6:1817–1853. [find](https://scholar.google.com/scholar?q=R%2E+Ando+and+T%2E+Zhang%2E+2005a%2E+A+framework+for+learning+predictive+structures+from+multiple+tasks+and+unlabeled+data%2E+JMLR%2C+6%3A1817%E2%80%931853%2E)
2. R. Ando and T. Zhang. 2005b. A high-performance semi-supervised learning method for text chunking. In ACL. [find](https://scholar.google.com/scholar?q=R%2E+Ando+and+T%2E+Zhang%2E+2005b%2E+A+high%2Dperformance+semi%2Dsupervised+learning+method+for+text+chunking%2E+In+ACL%2E)
3. R. Ando. 2004. Exploiting unannotated corpora for tagging and chunking. In ACL. Short paper. [find](https://scholar.google.com/scholar?q=R%2E+Ando%2E+2004%2E+Exploiting+unannotated+corpora+for+tagging+and+chunking%2E+In+ACL%2E+Short+paper%2E)
4. D. Blei, A. Ng, and M. Jordan. 2003. Latent dirichlet allocation. JMLR, 3:993–1022. [find](https://scholar.google.com/scholar?q=D%2E+Blei%2C+A%2E+Ng%2C+and+M%2E+Jordan%2E+2003%2E+Latent+dirichlet+allocation%2E+JMLR%2C+3%3A993%E2%80%931022%2E) [our card](https://md.research.elamur.ai/free/en/papers/blei-2003-latent-dirichlet-allocation/)
5. A. Blum and T. Mitchell. 1998. Combining labeled and unlabeled data with co-training. In Workshop on Computational Learning Theory. [find](https://scholar.google.com/scholar?q=A%2E+Blum+and+T%2E+Mitchell%2E+1998%2E+Combining+labeled+and+unlabeled+data+with+co%2Dtraining%2E+In+Workshop+on+Computational+Learning+Theory%2E) [our card](https://md.research.elamur.ai/free/en/papers/blum-1998-combining-labeled-unlabeled-data-co-training/)
6. P. Brown, V. Della Pietra, P. deSouza, J. Lai, and R. Mercer. 1992. Class-based n-gram models of natural language. Computational Linguistics, 18(4):467–479. [find](https://scholar.google.com/scholar?q=P%2E+Brown%2C+V%2E+Della+Pietra%2C+P%2E+deSouza%2C+J%2E+Lai%2C+and+R%2E+Mercer%2E+1992%2E+Class%2Dbased+n%2Dgram+models+of+natural+language%2E+Computational+Linguistics%2C+18%284%29%3A467%E2%80%93479%2E)
7. C. Chelba and A. Acero. 2004. Adaptation of maximum entropy capitalizer: Little data can help a lot. In EMNLP. [find](https://scholar.google.com/scholar?q=C%2E+Chelba+and+A%2E+Acero%2E+2004%2E+Adaptation+of+maximum+entropy+capitalizer%3A+Little+data+can+help+a+lot%2E+In+EMNLP%2E)
8. K. Crammer, Dekel O, J. Keshet, S. Shalev-Shwartz, and Y. Singer. 2006. Online passive-aggressive algorithms. JMLR, 7:551–585. [find](https://scholar.google.com/scholar?q=K%2E+Crammer%2C+Dekel+O%2C+J%2E+Keshet%2C+S%2E+Shalev%2DShwartz%2C+and+Y%2E+Singer%2E+2006%2E+Online+passive%2Daggressive+algorithms%2E+JMLR%2C+7%3A551%E2%80%93585%2E)
9. H. Daum´e III and D. Marcu. 2006. Domain adaptation for statistical classifiers. JAIR. [find](https://scholar.google.com/scholar?q=H%2E+Daum%C2%B4e+III+and+D%2E+Marcu%2E+2006%2E+Domain+adaptation+for+statistical+classifiers%2E+JAIR%2E)
10. R. Florian, H. Hassan, A.Ittycheriah, H. Jing, N. Kambhatla, X. Luo, N. Nicolov, and S. Roukos. 2004. A statistical model for multilingual entity detection and tracking. In of HLT-NAACL. [find](https://scholar.google.com/scholar?q=R%2E+Florian%2C+H%2E+Hassan%2C+A%2EIttycheriah%2C+H%2E+Jing%2C+N%2E+Kambhatla%2C+X%2E+Luo%2C+N%2E+Nicolov%2C+and+S%2E+Roukos%2E+2004%2E+A+statistical+model+for+multilingual+entity+detection+and+tracking%2E+In+of+HLT%2DNAACL%2E)
11. L. Gillick and S. Cox. 1989. Some statistical issues in the comparison of speech recognition algorithms. In ICASSP. [find](https://scholar.google.com/scholar?q=L%2E+Gillick+and+S%2E+Cox%2E+1989%2E+Some+statistical+issues+in+the+comparison+of+speech+recognition+algorithms%2E+In+ICASSP%2E)
12. R. Kuhn, P. Nguyen, J.C. Junqua, L. Goldwasser, N. Niedzielski, S. Fincke, K. Field, and M. Con-tolini. 1998. Eigenvoices for speaker adaptation. In ICSLP. [find](https://scholar.google.com/scholar?q=R%2E+Kuhn%2C+P%2E+Nguyen%2C+J%2EC%2E+Junqua%2C+L%2E+Goldwasser%2C+N%2E+Niedzielski%2C+S%2E+Fincke%2C+K%2E+Field%2C+and+M%2E+Con%2Dtolini%2E+1998%2E+Eigenvoices+for+speaker+adaptation%2E+In+ICSLP%2E)
13. M. Lease and E. Charniak. 2005. Parsing biomedical literature. In IJCNLP. [find](https://scholar.google.com/scholar?q=M%2E+Lease+and+E%2E+Charniak%2E+2005%2E+Parsing+biomedical+literature%2E+In+IJCNLP%2E)
14. M. Marcus, B. Santorini, and M. Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330. [find](https://scholar.google.com/scholar?q=M%2E+Marcus%2C+B%2E+Santorini%2C+and+M%2E+Marcinkiewicz%2E+1993%2E+Building+a+large+annotated+corpus+of+English%3A+The+Penn+Treebank%2E+Computational+Linguistics%2C+19%282%29%3A313%E2%80%93330%2E) [our card](https://md.research.elamur.ai/free/en/papers/bank-1993-marcus-building-large-annotated-corpus-english/)
15. R. McDonald, K. Crammer, and F. Pereira. 2005a. Flexible text segmentation with structured multilabel classification. In HLT-EMNLP. [find](https://scholar.google.com/scholar?q=R%2E+McDonald%2C+K%2E+Crammer%2C+and+F%2E+Pereira%2E+2005a%2E+Flexible+text+segmentation+with+structured+multilabel+classification%2E+In+HLT%2DEMNLP%2E)
16. R. McDonald, K. Crammer, and F. Pereira. 2005b. Online large-margin training of dependency parsers. In ACL. [find](https://scholar.google.com/scholar?q=R%2E+McDonald%2C+K%2E+Crammer%2C+and+F%2E+Pereira%2E+2005b%2E+Online+large%2Dmargin+training+of+dependency+parsers%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/narrative-2005-mcdonald-online-large-margin-training-dependency/)
17. S. Miller, J. Guinness, and A. Zamanian. 2004. Name tagging with word clusters and discriminative training. In HLT-NAACL. [find](https://scholar.google.com/scholar?q=S%2E+Miller%2C+J%2E+Guinness%2C+and+A%2E+Zamanian%2E+2004%2E+Name+tagging+with+word+clusters+and+discriminative+training%2E+In+HLT%2DNAACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/miller-2004-name-tagging-word-clusters-discriminative/)
18. F. Och. 2003. Minimum error rate training in statistical machine translation. In Proc. of ACL. [find](https://scholar.google.com/scholar?q=F%2E+Och%2E+2003%2E+Minimum+error+rate+training+in+statistical+machine+translation%2E+In+Proc%2E+of+ACL%2E)
19. PennBioIE. 2005. Mining The Bibliome Project. [http://bioie.ldc.upenn.edu/](http://bioie.ldc.upenn.edu/). [↗](http://bioie.ldc.upenn.edu/)
20. F. Pereira, N. Tishby, and L. Lee. 1993. Distributional clustering of english words. In ACL. [find](https://scholar.google.com/scholar?q=F%2E+Pereira%2C+N%2E+Tishby%2C+and+L%2E+Lee%2E+1993%2E+Distributional+clustering+of+english+words%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/pereira-1993-distributional-clustering-english-words/)
21. A. Ratnaparkhi. 1996. A maximum entropy model for part-of-speech tagging. In EMNLP. [find](https://scholar.google.com/scholar?q=A%2E+Ratnaparkhi%2E+1996%2E+A+maximum+entropy+model+for+part%2Dof%2Dspeech+tagging%2E+In+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/ratnaparkhi-1996-maximum-entropy-model-part-speech/)
22. B. Roark and M. Bacchiani. 2003. Supervised and unsupervised PCFG adaptation to novel domains. In HLT-NAACL. [find](https://scholar.google.com/scholar?q=B%2E+Roark+and+M%2E+Bacchiani%2E+2003%2E+Supervised+and+unsupervised+PCFG+adaptation+to+novel+domains%2E+In+HLT%2DNAACL%2E)
23. B. Roark, M. Saraclar, M. Collins, and M. Johnson. 2004. Discriminative language modeling with conditional random fields and the perceptron algorithm. In ACL. [find](https://scholar.google.com/scholar?q=B%2E+Roark%2C+M%2E+Saraclar%2C+M%2E+Collins%2C+and+M%2E+Johnson%2E+2004%2E+Discriminative+language+modeling+with+conditional+random+fields+and+the+perceptron+algorithm%2E+In+ACL%2E)
24. F. Sha and F. Pereira. 2003. Shallow parsing with conditional random fields. In HLT-NAACL. [find](https://scholar.google.com/scholar?q=F%2E+Sha+and+F%2E+Pereira%2E+2003%2E+Shallow+parsing+with+conditional+random+fields%2E+In+HLT%2DNAACL%2E)
25. K. Toutanova, D. Klein, C. D. Manning, and Y. Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In NAACL. [find](https://scholar.google.com/scholar?q=K%2E+Toutanova%2C+D%2E+Klein%2C+C%2E+D%2E+Manning%2C+and+Y%2E+Singer%2E+2003%2E+Feature%2Drich+part%2Dof%2Dspeech+tagging+with+a+cyclic+dependency+network%2E+In+NAACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/toutanova-2003-feature-rich-part-speech-tagging/)