---
title: "Rearranging the Familiar — paper by Loula et al. (2018)"
description: "The authors build on the SCAN benchmark (Lake and Baroni, 2018), in which simple navigation… 16 references, cited by 51 papers here."
authors: ["João Loula", "Marco Baroni", "Brenden Lake"]
year: 2018
venue: "Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP"
source: "https://aclanthology.org/W18-5413/"
doi: "10.18653/v1/w18-5413"
arxiv: "1807.07545"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/loula-2018-rearranging-familiar-testing-compositional-generalization/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/loula-2018-rearranging-familiar-testing-compositional-generalization/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/3/) / Rearranging the Familiar: Testing Compositional Generalization in Recurrent Networks

# Rearranging the Familiar: Testing Compositional Generalization in Recurrent Networks

**Authors:** João Loula, [Marco Baroni](https://md.research.elamur.ai/free/en/authors/marco-baroni/) and Brenden Lake · **Year:** 2018 · **Venue:** [Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP](https://md.research.elamur.ai/free/en/venues/emnlp-workshop-blackboxnlp-analyzing-and-interpreting-neural-networks-for-nlp/) ([EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP 2018](https://md.research.elamur.ai/free/en/venues/emnlp-workshop-blackboxnlp-analyzing-and-interpreting-neural-networks-for-nlp/2018/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/W18-5413/) · [PDF](https://aclanthology.org/W18-5413.pdf) · [DOI 10.18653/v1/w18-5413](https://doi.org/10.18653/v1/w18-5413) · [arXiv 1807.07545](https://arxiv.org/abs/1807.07545) · [OpenAlex](https://openalex.org/W2887970879)

**Citations:** 106 citations (OpenAlex, 2026-10-14) · [51](https://md.research.elamur.ai/free/en/papers/loula-2018-rearranging-familiar-testing-compositional-generalization/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary

The paper asks whether recurrent seq2seq models can generalize systematically by recombining words they already know well, as opposed to learning a brand-new word. The authors build on the SCAN benchmark (Lake and Baroni, 2018), in which simple navigation commands are translated into action sequences. They create new train/test splits in which every test word is frequent in training, but certain combinations, such as "jump around right", are withheld. Success therefore depends on exploiting the left/right symmetry and treating modifiers like "around" and "opposite" as second-order operators.

They reuse the strongest architecture from the original SCAN study, an LSTM seq2seq model, across three experiments. The model handles new combinations when the exact complex template has appeared with other fillers. It fails when it must derive a template's meaning from its parts, and it needs hundreds of in-context examples to reach peak accuracy.

The work shows that strong seq2seq generalization can come from analogy within familiar patterns rather than from learning compositional rules. This motivates research into inductive biases that would support systematicity.

## Key points

- **Dataset and setup:** SCAN has 12 base templates, combined through "and"/"after" and "twice"/"thrice" into 20,910 commands. The new splits are released in the SCAN GitHub repository.
- **Model and evaluation:**
  - The model is a 2-layer, 200-unit LSTM with 50% dropout. It scored 99.8% on SCAN's random split, and an attention variant did worse in every condition.
  - Accuracy counts only fully correct output sequences and is averaged over 5 runs.
  - Every condition uses 100k training presentations.
  - Commands whose target involves "turn" were excluded because "turn" has distinct semantics.
- **Experiment 1 (four held-out templates):**
  - Withholding only "jump around right" posed no difficulty.
  - Accuracy dropped sharply for more systematic gaps, including the shorter "Primitive right" pattern.
- **Experiment 2 (fillers of "Primitive around right"):** Seeing the template with only one other primitive, roughly 1,100 extra examples, was enough for near-perfect transfer to "jump".
- **Experiment 3 (1 to 1,024 "look around right" examples):** Accuracy climbed gradually, reaching about 70% at 64 examples and peaking at 512. This pattern suggests the model accumulates evidence case by case rather than extracting a rule.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Meta](https://md.research.elamur.ai/free/en/companies/meta/) — Company, Menlo Park, United States · 158 works · 211,459 citations
- [New York University](https://md.research.elamur.ai/free/en/universities/new-york-university/) — University, New York, United States · 136 works · 86,390 citations
- [École Polytechnique](https://md.research.elamur.ai/free/en/universities/ecole-polytechnique/) — University, Palaiseau, France · 5 works · 677 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

## Cite

Cite as: João Loula, Marco Baroni and Brenden Lake. 2018. Rearranging the Familiar: Testing Compositional Generalization in Recurrent Networks. Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{loula2018rearranging,
  title = {{Rearranging the Familiar: Testing Compositional Generalization in Recurrent Networks}},
  author = {João Loula, Marco Baroni and Brenden Lake},
  booktitle = {Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP},
  year = {2018},
  doi = {10.18653/v1/w18-5413},
  eprint = {1807.07545},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/w18-5413}
}
```

- [← previous Extreme Adaptation for Personalized Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/michel-2018-extreme-adaptation-personalized-neural-machine/)
- [next → Unsupervised Text Style Transfer using Language Models as Discriminators](https://md.research.elamur.ai/free/en/papers/yang-2018-unsupervised-text-style-transfer-using/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/3/)

## References

16 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Andrew Kyle Lampinen and James L McClelland. 2017. One-shot and few-shot learning of word embeddings. CoRR. [find](https://scholar.google.com/scholar?q=Andrew+Kyle+Lampinen+and+James+L+McClelland%2E+2017%2E+One%2Dshot+and+few%2Dshot+learning+of+word+embeddings%2E+CoRR%2E)
2. Matthew Botvinick and David Plaut. 2009. Empirical and computational support for context-dependent representations of serial order: Reply to Bowers, Damian, and Davis (2009). Psychological Review, 116(4):998–1002. [find](https://scholar.google.com/scholar?q=Matthew+Botvinick+and+David+Plaut%2E+2009%2E+Empirical+and+computational+support+for+context%2Ddependent+representations+of+serial+order%3A+Reply+to+Bowers%2C+Damian%2C+and+Davis+%282009%29%2E+Psychological+Review%2C+116)
3. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature, 521:436–444. [find](https://scholar.google.com/scholar?q=Yann+LeCun%2C+Yoshua+Bengio%2C+and+Geoffrey+Hinton%2E+2015%2E+Deep+learning%2E+Nature%2C+521%3A436%E2%80%93444%2E)
4. Gary F Marcus. 1998. Rethinking Eliminative Connectionism. Cognitive Psychology, 282(37):243–282. [find](https://scholar.google.com/scholar?q=Gary+F+Marcus%2E+1998%2E+Rethinking+Eliminative+Connectionism%2E+Cognitive+Psychology%2C+282%2837%29%3A243%E2%80%93282%2E)
5. Jeffrey Bowers, Markus Damian, and Colin David. 2009. A fundamental limitation of the conjunctive codes learned in PDP models of cognition: Comment on Botvinick and Plaut (2006). Psychological Review, 116(4):986–997. [find](https://scholar.google.com/scholar?q=Jeffrey+Bowers%2C+Markus+Damian%2C+and+Colin+David%2E+2009%2E+A+fundamental+limitation+of+the+conjunctive+codes+learned+in+PDP+models+of+cognition%3A+Comment+on+Botvinick+and+Plaut+%282006%29%2E+Psychological+Review%2C)
6. Gary F Marcus. 2003. The Algebraic Mind: Integrating Connectionism and Cognitive Science. MIT Press, Cambridge, MA. [find](https://scholar.google.com/scholar?q=Gary+F+Marcus%2E+2003%2E+The+Algebraic+Mind%3A+Integrating+Connectionism+and+Cognitive+Science%2E+MIT+Press%2C+Cambridge%2C+MA%2E)
7. Richard Montague. 1970. Universal grammar. Theoria, 36(3):373–398. [find](https://scholar.google.com/scholar?q=Richard+Montague%2E+1970%2E+Universal+grammar%2E+Theoria%2C+36%283%29%3A373%E2%80%93398%2E)
8. Philémon Brakel and Stefan Frank. 2009. Strong systematicity in sentence processing by simple recurrent networks. In Proceedings of CogSci, pages 1599–1604, Amsterdam, the Netherlands. [find](https://scholar.google.com/scholar?q=Phile%CC%81mon+Brakel+and+Stefan+Frank%2E+2009%2E+Strong+systematicity+in+sentence+processing+by+simple+recurrent+networks%2E+In+Proceedings+of+CogSci%2C+pages+1599%E2%80%931604%2C+Amsterdam%2C+the+Netherlands%2E)
9. Steven Phillips. 1998. Are feedforward and recurrent networks systematic? analysis and implications for a connectionist cognitive architecture. Connection Science, 10:137–160. [find](https://scholar.google.com/scholar?q=Steven+Phillips%2E+1998%2E+Are+feedforward+and+recurrent+networks+systematic%3F+analysis+and+implications+for+a+connectionist+cognitive+architecture%2E+Connection+Science%2C+10%3A137%E2%80%93160%2E)
10. Frankilin Chang. 2002. Symbolically speaking: a connectionist model of sentence production. Cognitive Science, 26:609–651. [find](https://scholar.google.com/scholar?q=Frankilin+Chang%2E+2002%2E+Symbolically+speaking%3A+a+connectionist+model+of+sentence+production%2E+Cognitive+Science%2C+26%3A609%E2%80%93651%2E)
11. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+Advances+in+neural+information+processing+systems%2C+pages+3104%E2%80%933112%2E)
12. Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. [On the properties of neural machine translation: Encoder-decoder approaches](https://md.research.elamur.ai/free/en/papers/cho-2014-properties-neural-machine-translation-encoder/). arXiv preprint arXiv:1409.1259.
13. Frank van der Velde, Gwendid van der Voort van der Kleij, and Marc de Kamps. 2004. Lack of combinatorial productivity in language processing with simple recurrent networks. Connection Science, 16(1):21–46. [find](https://scholar.google.com/scholar?q=Frank+van+der+Velde%2C+Gwendid+van+der+Voort+van+der+Kleij%2C+and+Marc+de+Kamps%2E+2004%2E+Lack+of+combinatorial+productivity+in+language+processing+with+simple+recurrent+networks%2E+Connection+Science%2C+16%281%29%3A2)
14. Noam Chomsky. 1957. Syntactic structures. Walter de Gruyter. [find](https://scholar.google.com/scholar?q=Noam+Chomsky%2E+1957%2E+Syntactic+structures%2E+Walter+de+Gruyter%2E)
15. Morten Christiansen and Nick Chater. 1994. Generalization and connectionist language learning. Mind & Language, 9(3):273–287. [find](https://scholar.google.com/scholar?q=Morten+Christiansen+and+Nick+Chater%2E+1994%2E+Generalization+and+connectionist+language+learning%2E+Mind+%26+Language%2C+9%283%29%3A273%E2%80%93287%2E)
16. Jerry A Fodor and Zenon W Pylyshyn. 1988. Connectionism and cognitive architecture: A critical analysis. Cognition, 28:3–71. [find](https://scholar.google.com/scholar?q=Jerry+A+Fodor+and+Zenon+W+Pylyshyn%2E+1988%2E+Connectionism+and+cognitive+architecture%3A+A+critical+analysis%2E+Cognition%2C+28%3A3%E2%80%9371%2E)

*Updated 24 October 2026*