---
title: "Jump to better conclusions: SCAN both left and right (2018)"
description: "As a complement, they introduce NACS, which reverses the direction and maps action sequences… 19 references, cited by 33 papers here."
authors: ["Jasmijn Bastings", "Marco Baroni", "Jason Weston", "Kyunghyun Cho", "Douwe Kiela"]
year: 2018
venue: "Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP"
source: "https://aclanthology.org/W18-5407/"
doi: "10.18653/v1/w18-5407"
arxiv: "1809.04640"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/bastings-2018-jump-better-conclusions-scan-both/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bastings-2018-jump-better-conclusions-scan-both/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/5/) / Jump to better conclusions: SCAN both left and right

# Jump to better conclusions: SCAN both left and right

**Authors:** [Jasmijn Bastings](https://md.research.elamur.ai/free/en/authors/jasmijn-bastings/), [Marco Baroni](https://md.research.elamur.ai/free/en/authors/marco-baroni/), [Jason Weston](https://md.research.elamur.ai/free/en/authors/jason-weston/), [Kyunghyun Cho](https://md.research.elamur.ai/free/en/authors/kyunghyun-cho/) and [Douwe Kiela](https://md.research.elamur.ai/free/en/authors/douwe-kiela/) · **Year:** 2018 · **Venue:** [Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP](https://md.research.elamur.ai/free/en/venues/emnlp-workshop-blackboxnlp-analyzing-and-interpreting-neural-networks-for-nlp/) ([EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP 2018](https://md.research.elamur.ai/free/en/venues/emnlp-workshop-blackboxnlp-analyzing-and-interpreting-neural-networks-for-nlp/2018/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/W18-5407/) · [PDF](https://aclanthology.org/W18-5407.pdf) · [DOI 10.18653/v1/w18-5407](https://doi.org/10.18653/v1/w18-5407) · [arXiv 1809.04640](https://arxiv.org/abs/1809.04640) · [OpenAlex](https://openalex.org/W2890835167)

**Citations:** 5 citations (OpenAlex, 2026-10-14) · [33](https://md.research.elamur.ai/free/en/papers/bastings-2018-jump-better-conclusions-scan-both/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
This BlackboxNLP 2018 paper re-examines SCAN, the command-to-action benchmark from Lake and Baroni (2018) that tests compositional generalization in recurrent sequence-to-sequence models. The authors argue that SCAN rewards overly simple models. Because its grammar is shallow, the outputs have almost no target-side dependencies. A decoder can therefore succeed by moving a pointer across the input and translating piece by piece, with no real composition.

As a complement, they introduce NACS, which reverses the direction and maps action sequences back to commands. This mapping is one-to-many and forces the model to keep track of what it has already generated.

They evaluate simple RNN, GRU and LSTM encoder-decoders, with and without Bahdanau attention. An ablation ("-Dep") cuts the decoder off from its previous outputs. They also rerun all of Lake and Baroni's experiments, including the English-French MT probe, using early stopping on held-out validation data.

Ablated models remain strong on SCAN but drop sharply on NACS. Even full attention models generalize poorly on NACS. The authors therefore present NACS as a more demanding benchmark for future work.

## Key points
- SCAN contains 20,910 unambiguous commands. It is tested in simple, length and primitive (turn left, jump) splits. For each split, 10% of training data was held out as a validation set for early stopping.
- On the simple split, every model scores roughly 100% on SCAN. Without access to previous outputs, NACS accuracy drops to 61.1 (RNN +Attn -Dep) and 51.2 (GRU +Attn -Dep), compared with 99.8 for the full models.
- On "turn left", the simpler -Dep attention models (92.0 and 90.8) beat their full counterparts on SCAN. On "jump", GRU +Attn reaches 12.5, compared with the 1.2 reported by L&B. Attention helped in every setting, and generalizing to longer sequences remained hard.
- In the "daxy" MT probe, a GRU with mini-batching and early stopping translated 70.8% (En-Fr) and 54.2% (Fr-En) of the novel sentences correctly. L&B had reported 12.5%.
- Code and data are available at github.com/facebookresearch/NACS.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Meta](https://md.research.elamur.ai/free/en/companies/meta/) — Company, Menlo Park, United States · 183 works · 228,558 citations
- [University of Amsterdam](https://md.research.elamur.ai/free/en/universities/university-of-amsterdam/) — University, Amsterdam, Netherlands · 144 works · 173,906 citations
- [Meta (United Kingdom)](https://md.research.elamur.ai/free/en/companies/meta-united-kingdom/) — Company, London, United Kingdom · 46 works · 18,033 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

## Cite

Cite as: Jasmijn Bastings, Marco Baroni, Jason Weston, Kyunghyun Cho and Douwe Kiela. 2018. Jump to better conclusions: SCAN both left and right. Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{bastings2018jump,
  title = {{Jump to better conclusions: SCAN both left and right}},
  author = {Jasmijn Bastings, Marco Baroni, Jason Weston, Kyunghyun Cho and Douwe Kiela},
  booktitle = {Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP},
  year = {2018},
  doi = {10.18653/v1/w18-5407},
  eprint = {1809.04640},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/w18-5407}
}
```

- [← previous Automatic Prediction of Discourse Connectives](https://md.research.elamur.ai/free/en/papers/malmi-2018-automatic-prediction-discourse-connectives/)
- [next → Generating Feedback for English Foreign Language Exercises](https://md.research.elamur.ai/free/en/papers/rudzewitz-2018-generating-feedback-english-foreign-language/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/5/)

## References

19 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. [Neural Machine Translation by Jointly Learning to Align and Translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). In Proceedings of the International Conference on Learning Representations (ICLR), San Diego, USA.
2. Adam Liska, Germán Kruszewski, and Marco Baroni. 2018. Memorize or generalize? searching for a compositional RNN in a haystack. CoRR, abs/1802.06467. — [arXiv:1802.06467](https://arxiv.org/abs/1802.06467) [find](https://scholar.google.com/scholar?q=Adam+Liska%2C+Germa%CC%81n+Kruszewski%2C+and+Marco+Baroni%2E+2018%2E+Memorize+or+generalize%3F+searching+for+a+compositional+RNN+in+a+haystack%2E+CoRR%2C+abs%2F1802%2E06467%2E+%E2%80%94+%5BarXiv%3A1802%2E06467%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1802%2E0)
3. Philémon Brakel and Stefan Frank. 2009. Strong systematicity in sentence processing by simple recurrent networks. In 31th Annual Conference of the Cognitive Science Society (COGSCI-2009), pages 1599–1604. Cognitive Science Society. [find](https://scholar.google.com/scholar?q=Phile%CC%81mon+Brakel+and+Stefan+Frank%2E+2009%2E+Strong+systematicity+in+sentence+processing+by+simple+recurrent+networks%2E+In+31th+Annual+Conference+of+the+Cognitive+Science+Society+%28COGSCI%2D2009%29%2C+pages+1599%E2%80%93)
4. Tomas Mikolov, Armand Joulin, and Marco Baroni. 2016. A roadmap towards machine intelligence. In International Conference on Intelligent Text Processing and Computational Linguistics, pages 29– 61. Springer. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Armand+Joulin%2C+and+Marco+Baroni%2E+2016%2E+A+roadmap+towards+machine+intelligence%2E+In+International+Conference+on+Intelligent+Text+Processing+and+Computational+Linguistics%2C+pages+29%E2%80%93+61%2E+Sp)
5. Steven Phillips. 1998. Are feedforward and recurrent networks systematic? analysis and implications for a connectionist cognitive architecture. Connection Science, 10(2):137–160. [find](https://scholar.google.com/scholar?q=Steven+Phillips%2E+1998%2E+Are+feedforward+and+recurrent+networks+systematic%3F+analysis+and+implications+for+a+connectionist+cognitive+architecture%2E+Connection+Science%2C+10%282%29%3A137%E2%80%93160%2E)
6. Rich Caruana, Steve Lawrence, and C Lee Giles. 2001. Overfitting in neural nets: Backpropagation, conjugate gradient, and early stopping. In Advances in neural information processing systems, pages 402– 408. [find](https://scholar.google.com/scholar?q=Rich+Caruana%2C+Steve+Lawrence%2C+and+C+Lee+Giles%2E+2001%2E+Overfitting+in+neural+nets%3A+Backpropagation%2C+conjugate+gradient%2C+and+early+stopping%2E+In+Advances+in+neural+information+processing+systems%2C+pages+40)
7. Paul Rodriguez and Janet Wiles. 1998. Recurrent neural networks can learn to implement symbol-sensitive counting. In Advances in Neural Information Processing Systems, pages 87–93. [find](https://scholar.google.com/scholar?q=Paul+Rodriguez+and+Janet+Wiles%2E+1998%2E+Recurrent+neural+networks+can+learn+to+implement+symbol%2Dsensitive+counting%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+87%E2%80%9393%2E)
8. Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Niki Parmar, Mike Schuster, Zhifeng Chen, et al. 2018. [The best of both worlds: Combining recent advances in neural machine translation](https://md.research.elamur.ai/free/en/papers/chen-2018-best-both-worlds-combining-recent/). arXiv preprint arXiv:1804.09849.
9. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. [Sequence to sequence learning with neural networks](https://md.research.elamur.ai/free/en/papers/sutskever-2014-sequence-sequence-learning-neural-networks/). In Advances in neural information processing systems, pages 3104–3112.
10. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. [Learning Phrase Representations using RNN Encoder– Decoder for Statistical Machine Translation](https://md.research.elamur.ai/free/en/papers/cho-2014-learning-phrase-representations-rnn-encoder/). In Proceedings of the 2014 Conference on Empirical Methods in Natural Langu…
11. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems, pages 6000–6010.
12. Gail Weiss, Yoav Goldberg, and Eran Yahav. 2018. [On the practical computational power of finite precision rnns for language recognition](https://md.research.elamur.ai/free/en/papers/weiss-2018-practical-computational-power-finite-precision/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Melbourne, Australia. Association for Computat…
13. Jeffrey L Elman. 1990. Finding structure in time. Cognitive science, 14(2):179–211. [find](https://scholar.google.com/scholar?q=Jeffrey+L+Elman%2E+1990%2E+Finding+structure+in+time%2E+Cognitive+science%2C+14%282%29%3A179%E2%80%93211%2E)
14. Jerry A Fodor and Zenon W Pylyshyn. 1988. Connectionism and cognitive architecture: A critical analysis. Cognition, 28(1-2):3–71. [find](https://scholar.google.com/scholar?q=Jerry+A+Fodor+and+Zenon+W+Pylyshyn%2E+1988%2E+Connectionism+and+cognitive+architecture%3A+A+critical+analysis%2E+Cognition%2C+28%281%2D2%29%3A3%E2%80%9371%2E)
15. Francis CK Wong and William SY Wang. 2007. Generalisation towards combinatorial productivity in language acquisition by simple recurrent networks. In Integration of Knowledge Intensive Multi-Agent Systems, 2007. [find](https://scholar.google.com/scholar?q=Francis+CK+Wong+and+William+SY+Wang%2E+2007%2E+Generalisation+towards+combinatorial+productivity+in+language+acquisition+by+simple+recurrent+networks%2E+In+Integration+of+Knowledge+Intensive+Multi%2DAgent+Sys)
16. KIMAS 2007. International Conference on, pages 139–144. IEEE. [find](https://scholar.google.com/scholar?q=KIMAS+2007%2E+International+Conference+on%2C+pages+139%E2%80%93144%2E+IEEE%2E)
17. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. [Convolutional sequence to sequence learning](https://md.research.elamur.ai/free/en/papers/gehring-2017-convolutional-sequence-sequence-learning/). arXiv preprint arXiv:1705.03122.
18. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation, 9(8):1735–1780. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+1997%2E+Long+Short%2DTerm+Memory%2E+Neural+Computation%2C+9%288%29%3A1735%E2%80%931780%2E)
19. Łukasz Kaiser and Samy Bengio. 2016. Can active memory replace attention? In Advances in Neural Information Processing Systems, pages 3781–3789. [find](https://scholar.google.com/scholar?q=%C5%81ukasz+Kaiser+and+Samy+Bengio%2E+2016%2E+Can+active+memory+replace+attention%3F+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+3781%E2%80%933789%2E)

*Updated 30 October 2026*