---
title: "Word Embeddings-Based Uncertainty Detection in Financial"
description: "The paper asks whether linguistic uncertainty in U.S. annual reports (10-Ks) can be… 87% accuracy. 21 references, cited by 7 papers here."
authors: ["Christoph Kilian Theil", "Sanja Štajner", "Heiner Stuckenschmidt"]
year: 2018
venue: "Proceedings of the First Workshop on Economics and Natural Language Processing"
source: "https://aclanthology.org/W18-3104/"
doi: "10.18653/v1/w18-3104"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/theil-2018-word-embeddings-based-uncertainty-detection/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/theil-2018-word-embeddings-based-uncertainty-detection/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/5/) / Word Embeddings-Based Uncertainty Detection in Financial Disclosures

# Word Embeddings-Based Uncertainty Detection in Financial Disclosures

**Authors:** Christoph Kilian Theil, [Sanja Štajner](https://md.research.elamur.ai/free/en/authors/sanja-stajner/) and [Heiner Stuckenschmidt](https://md.research.elamur.ai/free/en/authors/heiner-stuckenschmidt/) · **Year:** 2018 · **Venue:** Proceedings of the First Workshop on Economics and Natural Language Processing · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/W18-3104/) · [PDF](https://aclanthology.org/W18-3104.pdf) · [DOI 10.18653/v1/w18-3104](https://doi.org/10.18653/v1/w18-3104) · [OpenAlex](https://openalex.org/W2951416252)

**Citations:** 14 citations (OpenAlex, 2026-10-14) · [7](https://md.research.elamur.ai/free/en/papers/theil-2018-word-embeddings-based-uncertainty-detection/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper asks whether linguistic uncertainty in U.S. annual reports (10-Ks) can be detected better by enlarging a small, hand-made finance lexicon. The starting point is the Loughran and McDonald list of 297 uncertainty terms (Unc). For each term, the authors collected its 20 nearest neighbours by cosine similarity from two word2vec models. One was the generic Google News model. The other they trained themselves on roughly 125k 10-Ks. A finance- and linguistics-literate annotator then pruned the candidates, giving semi-automatic variants.

They tested every lexicon in two ways. The first was a panel regression of post-filing stock return volatility on almost 77k 10-Ks. The second was sentence classification on REPORTS, a newly annotated set of 1,000 10-K sentences labelled certain or uncertain.

The domain-trained expansion (UncSpec) did best in both tests. Its volatility coefficient was about double that of Unc, and it significantly improved precision and F1 for the uncertain class. Expert pruning gave little or no benefit. The practical lesson is that cheap embeddings trained on in-domain text can enrich specialised lexicons without much manual work.

## Key points
- **Corpus:**
  - 220,565 10-Ks (1994–2015) were downloaded from EDGAR, leaving 203,321 after removing duplicates and very short filings.
  - The filings were split into three parts: embedding training (124,830 filings, about 2.3B words), regressions (76,991) and classification sampling (1,500).
- **Lexicon sizes:**
  - Automatic expansions: UncGen has 2,333 terms, UncSpec 4,170, and their union 5,748.
  - Expert-filtered versions: UncGen exp has 538 terms, UncSpec exp 475, and their union 652.
  - Only 458 added terms were shared between the generic and domain expansions. After filtering, 241 were shared.
- **REPORTS dataset:**
  - Two annotators double-labelled a 100-sentence pilot and reached Cohen's κ = 0.73.
  - One annotator then labelled 900 more sentences. The final set contains 870 certain and 130 uncertain sentences.
- **Regression:** volatility was measured as market-model RMSE after filing, with year and Fama–French 48-industry fixed effects. The standardized coefficient rose from 0.016 for Unc to 0.034 for UncSpec (p ≤ 0.001).
- **Classification:**
  - Six Weka learners were compared, and JRip with tf–idf features was the strongest.
  - UncSpec exp reached F1 0.59, against 0.54 for Unc.
  - UncSpec reached the top precision of 0.69.
  - The majority-class baseline had 87% accuracy and no uncertain-class recall.

## Interesting ideas
- Growing a small seed lexicon by taking top-k neighbours from embeddings trained on in-domain text captures domain and legal wording (e.g. "more-likely-than-not") that generic models miss.
- Distributional neighbours mix antonyms with synonyms ("certainly" sits next to "probably"), so raw expansion can flip polarity.
- The authors separate two kinds of uncertainty words: markers of imprecision ("may", "somewhat") and words that merely name real-world risk ("volatility").
- A small double-annotated pilot with κ measured first, before a single annotator scales up, keeps labelling cheap but checked. Logging the disagreement cases also documents where the concept is genuinely blurry.
- A very strict expert filter cut coverage and did not help, because term weighting already dampened the noisy additions.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (world): Seed lists of hedges, modal markers or measure words in the compiled knowledge layer could be widened this way, with each added entry tagged by its seed and similarity so it stays explainable.
- **I2** (learn): Run each candidate through a WordNet antonym check as a negative control before it enters a lexicon, and reject entries that oppose their seed.
- **I3** (frames): In prag, treat a speaker's epistemic hedge as a property of the speech act, distinct from a frame whose content is about risk, so that "X is exposed to risk" is not read as the writer being unsure.
- **I4** (learn): Use the same pattern when building silver data, letting witness parsers or LLM tiers scale up only after agreement on a pilot slice, and keep the disputed items as a hard test set.
- **I5** (learn): When self-induced repair rules or lexicon entries are curated, measure recall loss from pruning against a held-out evaluation rather than assuming manual cleaning always improves quality.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Mannheim](https://md.research.elamur.ai/free/en/universities/university-of-mannheim/) — University, Mannheim, Germany · 18 works · 6,984 citations

📚 Shelves (8): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 744 works · [🎯 Decision Sciences](https://md.research.elamur.ai/free/en/shelves/decision-sciences/) · 85 works · [🤝 General Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/general-social-sciences/) · 87 works · [🎯 Management Science and Operations Research](https://md.research.elamur.ai/free/en/shelves/decision-sciences/management-science-and-operations-research/) · 36 works

## Cite

Cite as: Christoph Kilian Theil, Sanja Štajner and Heiner Stuckenschmidt. 2018. Word Embeddings-Based Uncertainty Detection in Financial Disclosures. Proceedings of the First Workshop on Economics and Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{theil2018word,
  title = {{Word Embeddings-Based Uncertainty Detection in Financial Disclosures}},
  author = {Christoph Kilian Theil, Sanja Štajner and Heiner Stuckenschmidt},
  booktitle = {Proceedings of the First Workshop on Economics and Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/w18-3104},
  url = {https://doi.org/10.18653/v1/w18-3104}
}
```

- [← previous Towards Coreference for Literary Text: Analyzing Domain-Specific Phenomena](https://md.research.elamur.ai/free/en/papers/roesiger-2018-coreference-literary-text-analyzing-domain/)
- [next → Gendered associations of English morphology](https://md.research.elamur.ai/free/en/papers/pierrehumbert-2018-gendered-associations-english-morphology/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/5/)

## References

21 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. J. Ross Quinlan. 1993. [C4.5: Programs for Machine Learning](https://md.research.elamur.ai/free/en/papers/quinlan-1993-c45-programs-machine-learning/). Morgan Kaufmann Publishers, San Francisco.
2. Johannes Amtén. 2014. Neural network plugin for Weka. [https://github.com/amten/NeuralNetwork](https://github.com/amten/NeuralNetwork). Accessed on January 16, 2018. [↗](https://github.com/amten/NeuralNetwork)
3. William Sharpe. 1963. A simplified model for portfolio analysis. Management Science, 9:277–293. [↗](https://doi.org/10.1287/mnsc.9.2.277)
4. Leo Breiman. 2001. Random forests. Machine Learning, 45(1):5–32. [find](https://scholar.google.com/scholar?q=Leo+Breiman%2E+2001%2E+Random+forests%2E+Machine+Learning%2C+45%281%29%3A5%E2%80%9332%2E)
5. Christoph Kilian Theil, Sanja Štajner, Heiner Stuckenschmidt, and Simone Paolo Ponzetto. 2017. Automatic detection of uncertain statements in the financial domain. In Lecture Notes in Computer Science: Proceedings of the CICLing, Berlin. Springer. In press. [find](https://scholar.google.com/scholar?q=Christoph+Kilian+Theil%2C+Sanja+S%CC%8Ctajner%2C+Heiner+Stuckenschmidt%2C+and+Simone+Paolo+Ponzetto%2E+2017%2E+Automatic+detection+of+uncertain+statements+in+the+financial+domain%2E+In+Lecture+Notes+in+Computer+Scienc)
6. Saskia le Cessie and Hans C. van Houwelingen. 1992. Ridge estimators in logistic regression. Applied Statistics, 41(1):191–201. [↗](https://doi.org/10.2307/2347628)
7. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):41–48. [↗](https://doi.org/10.1177/001316446002000104)
8. Ming-Feng Tsai and Chuan-Ju Wang. 2014. Financial keyword expansion via continuous word vector representations. Proceedings of the EMNLP, pages 1453–1458. [find](https://scholar.google.com/scholar?q=Ming%2DFeng+Tsai+and+Chuan%2DJu+Wang%2E+2014%2E+Financial+keyword+expansion+via+continuous+word+vector+representations%2E+Proceedings+of+the+EMNLP%2C+pages+1453%E2%80%931458%2E)
9. William W. Cohen. 1995. Fast effective rule induction. In Proceedings of the ICML, pages 115–123. [↗](https://doi.org/10.1016/b978-1-55860-377-6.50023-2)
10. Eugene F. Fama and Kenneth R. French. 1997. Industry costs of equity. Journal of Financial Economics, 43:153–193. [↗](https://doi.org/10.1016/s0304-405x(96)00896-3)
11. Sanja Štajner, Goran Glavaš, Simone Paolo Ponzetto, and Heiner Stuckenschmidt. 2017. Domain adaptation for automatic detection of speculative sentences. In Proceedings of the IEEE ICSC, pages 164–171. [find](https://scholar.google.com/scholar?q=Sanja+S%CC%8Ctajner%2C+Goran+Glavas%CC%8C%2C+Simone+Paolo+Ponzetto%2C+and+Heiner+Stuckenschmidt%2E+2017%2E+Domain+adaptation+for+automatic+detection+of+speculative+sentences%2E+In+Proceedings+of+the+IEEE+ICSC%2C+pages+164%E2%80%9317)
12. Viola Ganter and Michael Strube. 2009. [Finding hedges by chasing weasels: Hedge detection using Wikipedia tags and shallow linguistic features](https://md.research.elamur.ai/free/en/papers/ganter-2009-finding-hedges-chasing-weasels-hedge/). In Proceedings of the ACL–IJCNLP: Short Papers, pages 173–176.
13. Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H. Witten. 2009. The weka data mining software: An update. SIGKDD Explorations Newsletter, 11:10–18. [↗](https://doi.org/10.1145/1656274.1656278)
14. J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159–174. [↗](https://doi.org/10.2307/2529310)
15. Tim Loughran and Bill McDonald. 2011. When is a liability not a liability? Textual analysis, dictionaries, and 10-Ks. The Journal of Finance, 66(1):35–65. [find](https://scholar.google.com/scholar?q=Tim+Loughran+and+Bill+McDonald%2E+2011%2E+When+is+a+liability+not+a+liability%3F+Textual+analysis%2C+dictionaries%2C+and+10%2DKs%2E+The+Journal+of+Finance%2C+66%281%29%3A35%E2%80%9365%2E)
16. Tim Loughran and Bill McDonald. 2014. Measuring readability in financial disclosures. The Journal of Finance, 69(4):1643–1671. [find](https://scholar.google.com/scholar?q=Tim+Loughran+and+Bill+McDonald%2E+2014%2E+Measuring+readability+in+financial+disclosures%2E+The+Journal+of+Finance%2C+69%284%29%3A1643%E2%80%931671%2E)
17. Tim Loughran and Bill McDonald. 2016. Textual analysis in accounting and finance: A survey. Journal of Accounting Research, 54(4):1187–1230. [find](https://scholar.google.com/scholar?q=Tim+Loughran+and+Bill+McDonald%2E+2016%2E+Textual+analysis+in+accounting+and+finance%3A+A+survey%2E+Journal+of+Accounting+Research%2C+54%284%29%3A1187%E2%80%931230%2E)
18. Laurens J. P. van der Maaten and Geoffrey E. Hinton. 2008. Visualizing high-dimensional data using t-SNE. Journal of Machine Learning Research, 9:2579–2605. [find](https://scholar.google.com/scholar?q=Laurens+J%2E+P%2E+van+der+Maaten+and+Geoffrey+E%2E+Hinton%2E+2008%2E+Visualizing+high%2Ddimensional+data+using+t%2DSNE%2E+Journal+of+Machine+Learning+Research%2C+9%3A2579%E2%80%932605%2E)
19. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. [Efficient estimation of word representations in vector space](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/). ArXiv e-prints. [↗](https://doi.org/10.48550/arxiv.1301.3781)
20. Saif Mohammed, Bonnie Dorr, and Graeme Hirst. 2008. Computing word-pair antonymy. Proceedings of the EMNLP, pages 982–991. [find](https://scholar.google.com/scholar?q=Saif+Mohammed%2C+Bonnie+Dorr%2C+and+Graeme+Hirst%2E+2008%2E+Computing+word%2Dpair+antonymy%2E+Proceedings+of+the+EMNLP%2C+pages+982%E2%80%93991%2E)
21. John C. Platt. 1999. Fast training of support vector machines using sequential minimal optimization. In Advances in Kernel Methods—Support Vector Learning. [↗](https://doi.org/10.7551/mitpress/1130.003.0016)

*Updated 24 October 2026*