---
title: "ニューラル記号推論における推論過程の教示方法 — 青木洋一 2023"
description: "The authors investigate how to train a neural seq2seq model to write out intermediate reasoning steps so that it reliably solves formal multi-step…"
authors: ["青木洋一", "工藤慧音", "Ana Brassard", "栗林樹生", "吉川将司", "坂口慶祐", "乾健太郎"]
year: 2023
venue: "言語処理学会 第29回年次大会 発表論文集 (NLP2023)"
source: "https://www.anlp.jp/proceedings/annual_meeting/2023/#A5-4"
licence: "Free access (anlp.jp), no open licence stated — readonly; © the authors / 言語処理学会"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/aoki-2023-how-teach-reasoning-process/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/aoki-2023-how-teach-reasoning-process/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2023](https://md.research.elamur.ai/free/papers/2023/page/7/) / ニューラル記号推論における推論過程の教示方法

# ニューラル記号推論における推論過程の教示方法

**Authors:** 青木洋一, 工藤慧音, Ana Brassard, 栗林樹生, 吉川将司, 坂口慶祐, 乾健太郎 · **Year:** 2023 · **Venue:** 言語処理学会 第29回年次大会 発表論文集 (NLP2023) · **Licence:** Free access (anlp.jp), no open licence stated — readonly; © the authors / 言語処理学会

**Links:** [Publisher's page](https://www.anlp.jp/proceedings/annual_meeting/2023/#A5-4) · [PDF](https://www.anlp.jp/proceedings/annual_meeting/2023/pdf_dir/A5-4.pdf)

## Summary
The authors investigate how to train a neural seq2seq model to write out intermediate reasoning steps so that it reliably solves formal multi-step problems. To this end, they generated synthetic problems built from chains of variables. Each equation is either an assignment or an addition modulo 100, and the problems also include extra equations that distract the model. In such problems, the reasoning depth can be controlled precisely. The way intermediate steps are presented was split into two dimensions. The first is the output strategy: the whole chain at once, one step at a time, or one token at a time. The second is the reasoning strategy: the shortest path only, an exhaustive enumeration of the equations, or a backward search from the question followed by the shortest path. A T5-base model was trained on depths 1–5 and tested on depths up to 12. The results show that the format strongly determines whether the model can handle deeper problems. The best results come from step-by-step generation combined with either exhaustive enumeration or backward search: accuracy is nearly perfect even at depths not seen during training. Thus, at least under controlled conditions, a neural network can reliably perform symbolic reasoning if the training method is chosen correctly. This is a useful reference point for approaches such as chain-of-thought.

## Key points
- **Data:** the model was first trained for 30 epochs on 10K examples of basic operations (assignment, variable reference, addition). This was followed by 2000 epochs of training on 5K problems of depth 1–5, 1K for each depth. For evaluation, 200 examples were used for each depth from 1 to 12. Results are averaged over three seeds.
- **Output strategies:** generating the whole chain at once performs worst. At depth 12, the model still outputs chains that are too short, as if it had adapted to the lengths in the training data. The best option is step-by-step output, where each step is appended to the input. Token-by-token output falls behind step-by-step output. Generating intermediate steps in any format is better than answering directly.
- **Reasoning strategies** (with step-by-step output): exhaustive enumeration and backward search plus shortest path achieve 100% accuracy at depths 6–12, both for the answer and for the chain of steps. The shortest path alone achieves 92.5% at depth 12. The authors attribute this to the fact that this format hides from the model the very stage of searching for the relevant equations.
- **Error analysis** (32 incorrect answers at depth 12, shortest-path strategy): 53% are inaccurate copying of an equation from the problem statement. The second type of error is premature substitution, where the model skips the copying step and substitutes a value at random. This is an argument in favor of copy mechanisms.
- **Additional experiments:** T5-large noticeably improves only token-by-token output. T5 generally outperforms BART-base, which the authors link to T5's pre-training on NLP tasks.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{2023,
  title = {{ニューラル記号推論における推論過程の教示方法}},
  author = {青木洋一 and 工藤慧音 and Brassard, Ana and 栗林樹生 and 吉川将司 and 坂口慶祐 and 乾健太郎},
  booktitle = {言語処理学会 第29回年次大会 発表論文集 (NLP2023)},
  year = {2023},
  url = {https://www.anlp.jp/proceedings/annual_meeting/2023/#A5-4}
}
```

- [← previous ÚFAL CorPipe at CRAC 2023: Larger Context Improves Multilingual Coreference Resolution](https://md.research.elamur.ai/free/papers/straka-2023-fal-corpipe-crac-2023-larger/)
- [next → 一种改进的汉语语义角色分类体系与标注实践](https://md.research.elamur.ai/free/papers/song-2023-fine-grained-annotated/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/papers/2023/page/7/)