---
title: "Expanding the Scope of the ATIS Task: The ATIS-3 Corpus (1994)"
description: "This paper reports on the third round of the ARPA Air Travel Information System (ATIS) benchmark… 6 references, cited by 134 papers here."
authors: ["Deborah A. Dahl", "Madeleine Bates", "Michael Brown", "William Fisher", "Kate Hunicke-Smith", "David Pallett", "Christine Pao", "Alexander Rudnicky", "Elizabeth Shriberg"]
year: 1994
venue: "Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994"
source: "https://aclanthology.org/H94-1010/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/dahl-1994-expanding-scope-atis-task-atis/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/dahl-1994-expanding-scope-atis-task-atis/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [1994](https://md.research.elamur.ai/free/en/papers/1994/) / Expanding the Scope of the ATIS Task: The ATIS-3 Corpus

# Expanding the Scope of the ATIS Task: The ATIS-3 Corpus

**Authors:** Deborah A. Dahl, Madeleine Bates, Michael Brown, William Fisher, Kate Hunicke-Smith, David Pallett, Christine Pao, [Alexander Rudnicky](https://md.research.elamur.ai/free/en/authors/alexander-rudnicky/) and [Elizabeth Shriberg](https://md.research.elamur.ai/free/en/authors/elizabeth-shriberg/) · **Year:** 1994 · **Venue:** [Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994](https://md.research.elamur.ai/free/en/venues/human-language-technology-proceedings-of-a-workshop-held-at-plainsboro-new/) ([Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11 1994](https://md.research.elamur.ai/free/en/venues/human-language-technology-proceedings-of-a-workshop-held-at-plainsboro-new/1994/)) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/H94-1010/) · [PDF](https://aclanthology.org/H94-1010.pdf) · [OpenAlex](https://openalex.org/W7098591660)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [134](https://md.research.elamur.ai/free/en/papers/dahl-1994-expanding-scope-atis-task-atis/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
This paper reports on the third round of the ARPA Air Travel Information System (ATIS) benchmark, coordinated by the MADCOW working group. Earlier ATIS corpora were collected against a small 11-city database, which seemed too unrealistic for maturing spoken language systems. The authors therefore built a much larger relational database from June 1992 Official Airline Guide (OAG) data and used it to gather the ATIS-3 corpus at five sites (BBN, CMU, MIT, NIST, SRI). Collection was mostly automatic: the sites' own systems handled recognition and interpretation, except MIT, which used a human transcription wizard. SRI annotated utterances as context-independent (A), context-dependent (D), or unevaluable (X), following the Principles of Interpretation. The paper gives per-site breakdowns and notes that the A/D/X proportions vary widely, probably because each site used different scenarios. A key finding is that the roughly thirtyfold increase in flights caused no serious problems for collection, evaluation or annotation. This suggests that systems developed on small databases can scale. The paper also outlines plans for a development test set, alternative evaluations and telephone-based successor tasks.

## Key points
- **Database size:** the expanded database covers 46 cities, 52 airports and 23,457 flights. The old one had 11 cities, 9 airports and 765 flights.
- **Corpus size:** ATIS-3 contains 12,047 utterances, of which 3,876 are annotated. This splits into 8,297 released training utterances (2,906 annotated) and 3,211 held out for testing.
- **Held-out and unreleased data:**
  - About 20% of submitted data goes to the evaluation test pool.
  - 500 NIST utterances are reserved for a semantic-evaluation dry run.
  - 1,440 CMU utterances had not yet been released.
- **Session files:** each session comes with .wav, .log, .sro (detailed transcription), .cat (A/D/X class), .ref (minimal answer) and .rf2 (maximal answer).
- **Planned development test set:** although nearly 25,000 ATIS utterances existed by then, there was no standard dev set. MADCOW planned to collect 300–500 utterances for one in spring 1994.

## Interesting ideas
- **Min/max reference answers:** storing both a minimal and a maximal correct answer allows tolerant scoring. A system is neither punished for adding reasonable extra information nor allowed to over-answer without limit. This could carry over to QA or text-to-SQL benchmarks.
- **Annotate test data late:** test data is labeled just before the evaluation, so it matches the latest interpretation guidelines. This prevents guideline drift from making the gold labels inconsistent.
- **Keep a dedicated, untouched dev set:** after adjudication, developers have studied the evaluation test data closely, so it is contaminated for later development use. A separate, unseen dev set enables fair comparisons across sites and across years.
- **Explicit "unevaluable" class:** marking queries that cannot be scored (Class X) keeps the metrics clean. Separating context-independent from context-dependent queries also shows where dialog context matters.
- **Within-subjects design for task-level evaluations:** in end-to-end, human-judged evaluations, a within-subjects design reduces outside variation. That makes statistically reliable results achievable with fewer participants.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

## Cite

Cite as: Deborah A. Dahl, Madeleine Bates, Michael Brown, William Fisher, Kate Hunicke-Smith, David Pallett, Christine Pao, Alexander Rudnicky and Elizabeth Shriberg. 1994. Expanding the Scope of the ATIS Task: The ATIS-3 Corpus. Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{dahl1994expanding,
  title = {{Expanding the Scope of the ATIS Task: The ATIS-3 Corpus}},
  author = {Deborah A. Dahl, Madeleine Bates, Michael Brown, William Fisher, Kate Hunicke-Smith, David Pallett, Christine Pao, Alexander Rudnicky and Elizabeth Shriberg},
  booktitle = {Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994},
  year = {1994},
  url = {https://aclanthology.org/H94-1010/}
}
```

- [← previous Construction of a Bilingual Dictionary Intermediated by a Third Language](https://md.research.elamur.ai/free/en/papers/tanaka-1994-bilingual-dictionary-third-language/)
- [next → Comlex Syntax: Building a Computational Lexicon](https://md.research.elamur.ai/free/en/papers/grishman-1994-comlex-syntax-building-computational-lexicon/)
- [↑ back to papers of 1994](https://md.research.elamur.ai/free/en/papers/1994/)

## References

6 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Price, E "Evaluation of spoken language systems: the ATIS domain". In Proceedings of the speech and natural language workshop. Morgan-Kaufmann, 1990. [find](https://scholar.google.com/scholar?q=Price%2C+E+%22Evaluation+of+spoken+language+systems%3A+the+ATIS+domain%22%2E+In+Proceedings+of+the+speech+and+natural+language+workshop%2E+Morgan%2DKaufmann%2C+1990%2E)
2. Hemphill, C. T., J. J. Godfrey, and G. R. Doddington. "The ATIS spoken language systems pilot corpus". In Proceedings of the speech and natural language workshop. Morgan Kaufmann, 1990. Figure 2: Class A, D, and X in Test Data Class A, D, and X in Test Data rances [] I , Class KI A [] Class A [] ,-1… [find](https://scholar.google.com/scholar?q=Hemphill%2C+C%2E+T%2E%2C+J%2E+J%2E+Godfrey%2C+and+G%2E+R%2E+Doddington%2E+%22The+ATIS+spoken+language+systems+pilot+corpus%22%2E+In+Proceedings+of+the+speech+and+natural+language+workshop%2E+Morgan+Kaufmann%2C+1990%2E+Figure+2%3A+Clas)
3. MADCOW. "Multi-site data collection for a spoken language corpus" Proceedings of the fifth DARPA speech and natural language workshop. Morgan Kaufmann, 1992. [find](https://scholar.google.com/scholar?q=MADCOW%2E+%22Multi%2Dsite+data+collection+for+a+spoken+language+corpus%22+Proceedings+of+the+fifth+DARPA+speech+and+natural+language+workshop%2E+Morgan+Kaufmann%2C+1992%2E)
4. Hirschman, L. M. Bates, D. Dahl, W. Fisher, D. Pallett, Kate Hunicke-Smith. P. Price, A. Rudnicky, and E. Tzoukermann."Multi-site data collection and evaluation in spoken language understanding". Proceedings of the Human Language Teclmology Workshop, March, 1993. [find](https://scholar.google.com/scholar?q=Hirschman%2C+L%2E+M%2E+Bates%2C+D%2E+Dahl%2C+W%2E+Fisher%2C+D%2E+Pallett%2C+Kate+Hunicke%2DSmith%2E+P%2E+Price%2C+A%2E+Rudnicky%2C+and+E%2E+Tzoukermann%2E%22Multi%2Dsite+data+collection+and+evaluation+in+spoken+language+understanding%22%2E+Proc)
5. Pallett, David, Jonathan Fiscus, William Fisher, John Garofolo, Bruce Lund, Mark Pryzbocki, "1993 B enclmaark Tests for the ARPA Spoken Language Program" (this volume). [find](https://scholar.google.com/scholar?q=Pallett%2C+David%2C+Jonathan+Fiscus%2C+William+Fisher%2C+John+Garofolo%2C+Bruce+Lund%2C+Mark+Pryzbocki%2C+%221993+B+enclmaark+Tests+for+the+ARPA+Spoken+Language+Program%22+%28this+volume%29%2E)
6. Moore, R. "Semantic Evaluation for Spoken Language Systems" (this volume). [find](https://scholar.google.com/scholar?q=Moore%2C+R%2E+%22Semantic+Evaluation+for+Spoken+Language+Systems%22+%28this+volume%29%2E)

*Updated 24 October 2026*