---
title: "言語モデルの思考連鎖的推論における探索戦略の動的変化 — 青木洋一 2024"
description: "The authors study how large language models choose the next step when reasoning in the chain-of-thought style."
authors: ["青木洋一", "工藤慧音", "曾根周作", "栗林樹生", "谷口雅弥", "坂口慶祐", "乾健太郎"]
year: 2024
venue: "言語処理学会 第30回年次大会 発表論文集 (NLP2024)"
source: "https://www.anlp.jp/proceedings/annual_meeting/2024/#A11-6"
licence: "Free access (anlp.jp), no open licence stated — readonly; © the authors / 言語処理学会"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/aoki-2024-dynamic-changes-search-strategy/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/aoki-2024-dynamic-changes-search-strategy/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2024](https://md.research.elamur.ai/free/papers/2024/page/4/) / 言語モデルの思考連鎖的推論における探索戦略の動的変化

# 言語モデルの思考連鎖的推論における探索戦略の動的変化

**Authors:** 青木洋一, 工藤慧音, 曾根周作, 栗林樹生, 谷口雅弥, 坂口慶祐, 乾健太郎 · **Year:** 2024 · **Venue:** 言語処理学会 第30回年次大会 発表論文集 (NLP2024) · **Licence:** Free access (anlp.jp), no open licence stated — readonly; © the authors / 言語処理学会

**Links:** [Publisher's page](https://www.anlp.jp/proceedings/annual_meeting/2024/#A11-6) · [PDF](https://www.anlp.jp/proceedings/annual_meeting/2024/pdf_dir/A11-6.pdf)

## Summary
The authors study how large language models choose the next step when reasoning in the chain-of-thought style. The material consists of multi-step arithmetic word problems in which some of the conditions are not needed for the answer. Two possible strategies were tested. Under the first, the model implicitly traces the shortest path from the conditions to the answer. Under the second, it relies on surface cues: the similarity of a sentence to the question, the absence of negation, and the sentence's position in the text. First, a single extra sentence carrying such a cue was added to GSM8K problems, and the authors counted how often GPT-3.5-turbo mentioned it in its reasoning. Then, on a synthetic problem set, GPT-3.5-turbo and GPT-4 were given part of the correct chain, and the authors observed what the model chose next: the needed condition or the "decoy". It turned out that the strategy changes over the course of solving. At the start, the models more often follow surface cues, and the closer they get to the answer, the more often they choose steps on the shortest path. GPT-4 chooses shortest-path steps more often than GPT-3.5, even at early stages. The work shows at which stage CoT reasoning is most easily derailed by irrelevant information.

## Key points
- **Data for experiment 1.** GSM8K (about 1K test problems) was taken, and six variants with an additional distractor sentence were created: GSM8K-overlap / not-overlap (the subject matches or does not match the subject of the question), GSM8K-pos / neg (the sentence without or with "not"), GSM8K-head / tail (the sentence at the beginning or at the end of the problem statement).
- **GPT-3.5-turbo results.** The model mentioned the distractor in 59.5% of cases versus 39.2% for overlap / not-overlap, in 27.0% versus 24.3% for pos / neg, and in 39.2% versus 27.0% for head / tail. Similarity of the sentence to the question has the strongest effect.
- **Experiment 2.** 100 synthetic problems about numbers of apples were created in a "formal language"; in each, the shortest path has length 3. The models received the first steps of the correct chain, and the distractor had the same subject as the question.
- **Dynamics.** In GPT-3.5 and GPT-4, the share of steps following a surface cue decreases with each step, while the share of shortest-path steps increases. GPT-4 finds the shortest path more confidently, even when there are still many options. The authors attribute this to the model's larger size.
- **Limitations.** Only models of the GPT family were tested. The conclusions are drawn solely from the models' behavior, without analysis of their internal representations. The paper was published at NLP2024.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{2024,
  title = {{言語モデルの思考連鎖的推論における探索戦略の動的変化}},
  author = {青木洋一 and 工藤慧音 and 曾根周作 and 栗林樹生 and 谷口雅弥 and 坂口慶祐 and 乾健太郎},
  booktitle = {言語処理学会 第30回年次大会 発表論文集 (NLP2024)},
  year = {2024},
  url = {https://www.anlp.jp/proceedings/annual_meeting/2024/#A11-6}
}
```

- [← previous 视频问答技术研究进展](https://md.research.elamur.ai/free/papers/bao-2024-video-question-answering/)
- [next → 語彙置換継続事前学習による日英バイリンガルモデルの構築と評価](https://md.research.elamur.ai/free/papers/asaba-2024-building-evaluating-japanese-english/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/papers/2024/page/4/)