---
title: "On the Automatic Generation of Medical Imaging Reports (2018)"
description: "To handle this, the paper builds a single multi-task model: A CNN encodes image regions. 26 references, cited by 60 papers here."
authors: ["Baoyu Jing", "Pengtao Xie", "Eric Xing"]
year: 2018
venue: "Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/P18-1240/"
doi: "10.18653/v1/p18-1240"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/jing-2018-automatic-generation-medical-imaging-reports/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/jing-2018-automatic-generation-medical-imaging-reports/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/) / On the Automatic Generation of Medical Imaging Reports

# On the Automatic Generation of Medical Imaging Reports

**Authors:** Baoyu Jing, Pengtao Xie and [Eric Xing](https://md.research.elamur.ai/free/en/authors/eric-xing/) · **Year:** 2018 · **Venue:** [Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2018](https://md.research.elamur.ai/free/en/venues/acl/2018/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/P18-1240/) · [PDF](https://aclanthology.org/P18-1240.pdf) · [DOI 10.18653/v1/p18-1240](https://doi.org/10.18653/v1/p18-1240) · [OpenAlex](https://openalex.org/W2770165365)

**Citations:** 420 citations (OpenAlex, 2026-10-14) · [60](https://md.research.elamur.ai/free/en/papers/jing-2018-automatic-generation-medical-imaging-reports/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Writing reports for medical images takes skilled physicians a lot of time and is error-prone for less experienced ones. The authors therefore study how to generate such reports automatically. A full report mixes different kinds of content, such as keyword tags and a multi-sentence findings paragraph. Abnormal regions are also hard to localize.

To handle this, the paper builds a single multi-task model:
- A CNN encodes image regions.
- A multi-label classifier predicts tags, and the embeddings of those tags serve as semantic features.
- A co-attention module weighs visual and semantic information together to form a context vector.
- A two-level (hierarchical) LSTM decodes the report. A sentence-level LSTM produces one topic vector per sentence and decides when to stop, and a word-level LSTM turns each topic into a sentence.

The model was tested on two public datasets of radiology and pathology images. The full co-attention model beat the baselines and its own ablated variants on every reported caption metric. Its output was also closest to the real balance of normal versus abnormal findings. This makes it an early step toward generating free-form clinical narratives rather than templated or semi-structured ones.

## Key points
- Reports have three parts: impression, findings and tags. The tags are keywords extracted with the Medical Text Indexer (MTI) and are predicted as a multi-label classification task.
- Visual features come from the last convolutional layer of VGG-19. Its last two fully connected layers act as the MLC tag predictor, and embeddings of the top-M tags feed the co-attention.
- The sentence LSTM uses deep output layers for both topic generation and a STOP/CONTINUE decision. The word LSTM then generates each sentence from its topic.
- Share of generated sentences describing abnormalities (Table 2):
  - Ours-CoAttention: 0.529
  - Soft Attention: 0.490
  - Ours-no-Attention: 0.247
  - Ground truth: 0.615
- Motivation: typing the findings for a single image takes about 5–10 minutes. The authors present this as the first system to generate natural, physician-written-style reports rather than semi-structured ones.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 365 works · 158,048 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works · [🩺 Medicine](https://md.research.elamur.ai/free/en/shelves/medicine/) · 80 works · [🩺 Health Informatics](https://md.research.elamur.ai/free/en/shelves/medicine/health-informatics/) · 53 works

## Cite

Cite as: Baoyu Jing, Pengtao Xie and Eric Xing. 2018. On the Automatic Generation of Medical Imaging Reports. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{jing2018automatic,
  title = {{On the Automatic Generation of Medical Imaging Reports}},
  author = {Baoyu Jing, Pengtao Xie and Eric Xing},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-1240},
  url = {https://doi.org/10.18653/v1/p18-1240}
}
```

- [← previous Legal Judgment Prediction via Topological Learning](https://md.research.elamur.ai/free/en/papers/zhong-2018-legal-judgment-prediction-topological-learning/)
- [next → Loss in Translation: Learning Bilingual Word Mapping with a Retrieval Criterion](https://md.research.elamur.ai/free/en/papers/joulin-2018-loss-translation-learning-bilingual-word/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/)

## References

26 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jimmy Ba, Volodymyr Mnih, and Koray Kavukcuoglu. 2015. Multiple object recognition with visual attention. ICLR. [find](https://scholar.google.com/scholar?q=Jimmy+Ba%2C+Volodymyr+Mnih%2C+and+Koray+Kavukcuoglu%2E+2015%2E+Multiple+object+recognition+with+visual+attention%2E+ICLR%2E)
2. Louke Delrue, Robert Gosselin, Bart Ilsen, An Van Landeghem, Johan de Mey, and Philippe Duyck. 2011. Difficulties in the interpretation of chest radiography. In Comparative Interpretation of CT and Standard Radiography of the Chest, pages 27–49. Springer. [find](https://scholar.google.com/scholar?q=Louke+Delrue%2C+Robert+Gosselin%2C+Bart+Ilsen%2C+An+Van+Landeghem%2C+Johan+de+Mey%2C+and+Philippe+Duyck%2E+2011%2E+Difficulties+in+the+interpretation+of+chest+radiography%2E+In+Comparative+Interpretation+of+CT+and+St)
3. Dina Demner-Fushman, Marc D Kohli, Marc B Rosenman, Sonya E Shooshan, Laritza Rodriguez, Sameer Antani, George R Thoma, and Clement J McDonald. 2015. Preparing a collection of radiology examinations for distribution and retrieval. Journal of the American Medical Informatics Association, 23(2):304–31… [find](https://scholar.google.com/scholar?q=Dina+Demner%2DFushman%2C+Marc+D+Kohli%2C+Marc+B+Rosenman%2C+Sonya+E+Shooshan%2C+Laritza+Rodriguez%2C+Sameer+Antani%2C+George+R+Thoma%2C+and+Clement+J+McDonald%2E+2015%2E+Preparing+a+collection+of+radiology+examinations+f)
4. Michael Denkowski and Alon Lavie. 2014. [Meteor universal: Language specific translation evaluation for any target language](https://md.research.elamur.ai/free/en/papers/denkowski-2014-meteor-universal-language-specific-translation/). In Proceedings of the ninth workshop on statistical machine translation, pages 376–380.
5. Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. 2015. [Long-term recurrent convolutional networks for visual recognition and description](https://md.research.elamur.ai/free/en/papers/donahue-2014-long-term-recurrent-convolutional-networks/). In Proceedings of the IEEE conference on computer vision and pattern recognition…
6. Hao Fang, Saurabh Gupta, Forrest Iandola, Rupesh K Srivastava, Li Deng, Piotr Dollár, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C Platt, et al. 2015. From captions to visual concepts and back. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1473–1482… [find](https://scholar.google.com/scholar?q=Hao+Fang%2C+Saurabh+Gupta%2C+Forrest+Iandola%2C+Rupesh+K+Srivastava%2C+Li+Deng%2C+Piotr+Dolla%CC%81r%2C+Jianfeng+Gao%2C+Xiaodong+He%2C+Margaret+Mitchell%2C+John+C+Platt%2C+et+al%2E+2015%2E+From+captions+to+visual+concepts+and+bac)
7. Yunchao Gong, Yangqing Jia, Thomas Leung, Alexander Toshev, and Sergey Ioffe. 2013. Deep convolutional ranking for multilabel image annotation. ICLR. [find](https://scholar.google.com/scholar?q=Yunchao+Gong%2C+Yangqing+Jia%2C+Thomas+Leung%2C+Alexander+Toshev%2C+and+Sergey+Ioffe%2E+2013%2E+Deep+convolutional+ranking+for+multilabel+image+annotation%2E+ICLR%2E)
8. Matthieu Guillaumin, Thomas Mensink, Jakob Verbeek, and Cordelia Schmid. 2009. Tagprop: Discriminative metric learning in nearest neighbor models for image auto-annotation. In Computer Vision, 2009 IEEE 12th International Conference on, pages 309–316. IEEE. [find](https://scholar.google.com/scholar?q=Matthieu+Guillaumin%2C+Thomas+Mensink%2C+Jakob+Verbeek%2C+and+Cordelia+Schmid%2E+2009%2E+Tagprop%3A+Discriminative+metric+learning+in+nearest+neighbor+models+for+image+auto%2Dannotation%2E+In+Computer+Vision%2C+2009+IE)
9. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A1735%E2%80%931780%2E)
10. Andrej Karpathy and Li Fei-Fei. 2015. [Deep visual-semantic alignments for generating image descriptions](https://md.research.elamur.ai/free/en/papers/karpathy-2014-deep-visual-semantic-alignments-generating/). In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3128–3137.
11. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. [find](https://scholar.google.com/scholar?q=Diederik+P+Kingma+and+Jimmy+Ba%2E+2014%2E+Adam%3A+A+method+for+stochastic+optimization%2E+arXiv+preprint+arXiv%3A1412%2E6980%2E)
12. Pavel Kisilev, Eugene Walach, Ella Barkan, Boaz Ophir, Sharon Alpert, and Sharbell Y Hashoul. 2015. From medical image to automatic medical report generation. IBM Journal of Research and Development, 59(2/3):2–1. [find](https://scholar.google.com/scholar?q=Pavel+Kisilev%2C+Eugene+Walach%2C+Ella+Barkan%2C+Boaz+Ophir%2C+Sharon+Alpert%2C+and+Sharbell+Y+Hashoul%2E+2015%2E+From+medical+image+to+automatic+medical+report+generation%2E+IBM+Journal+of+Research+and+Development%2C+)
13. Jonathan Krause, Justin Johnson, Ranjay Krishna, and Li Fei-Fei. 2017. A hierarchical approach for generating descriptive image paragraphs. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). [find](https://scholar.google.com/scholar?q=Jonathan+Krause%2C+Justin+Johnson%2C+Ranjay+Krishna%2C+and+Li+Fei%2DFei%2E+2017%2E+A+hierarchical+approach+for+generating+descriptive+image+paragraphs%2E+In+The+IEEE+Conference+on+Computer+Vision+and+Pattern+Recogn)
14. Xiaodan Liang, Zhiting Hu, Hao Zhang, Chuang Gan, and Eric P. Xing. 2017. Recurrent topic-transition gan for visual paragraph generation. In The IEEE International Conference on Computer Vision (ICCV). [find](https://scholar.google.com/scholar?q=Xiaodan+Liang%2C+Zhiting+Hu%2C+Hao+Zhang%2C+Chuang+Gan%2C+and+Eric+P%2E+Xing%2E+2017%2E+Recurrent+topic%2Dtransition+gan+for+visual+paragraph+generation%2E+In+The+IEEE+International+Conference+on+Computer+Vision+%28ICCV%29)
15. Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. 2016. Image captioning with semantic attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4651–4659. [find](https://scholar.google.com/scholar?q=Quanzeng+You%2C+Hailin+Jin%2C+Zhaowen+Wang%2C+Chen+Fang%2C+and+Jiebo+Luo%2E+2016%2E+Image+captioning+with+semantic+attention%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+page)
16. Chin-Yew Lin. 2004. [Rouge: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text summarization branches out: Proceedings of the ACL-04 workshop, volume 8. Barcelona, Spain.
17. Zizhao Zhang, Yuanpu Xie, Fuyong Xing, Mason Mc-Gough, and Lin Yang. 2017. Mdnet: A semantically and visually interpretable medical image diagnosis network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6428–6436. [find](https://scholar.google.com/scholar?q=Zizhao+Zhang%2C+Yuanpu+Xie%2C+Fuyong+Xing%2C+Mason+Mc%2DGough%2C+and+Lin+Yang%2E+2017%2E+Mdnet%3A+A+semantically+and+visually+interpretable+medical+image+diagnosis+network%2E+In+Proceedings+of+the+IEEE+Conference+on+Co)
18. Pengfei Liu, Xipeng Qiu, Xinchi Chen, Shiyu Wu, and Xuanjing Huang. 2015. Multi-timescale long short-term memory neural network for modelling sentences and documents. In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 2326–2335. [find](https://scholar.google.com/scholar?q=Pengfei+Liu%2C+Xipeng+Qiu%2C+Xinchi+Chen%2C+Shiyu+Wu%2C+and+Xuanjing+Huang%2E+2015%2E+Multi%2Dtimescale+long+short%2Dterm+memory+neural+network+for+modelling+sentences+and+documents%2E+In+Proceedings+of+the+2015+confer)
19. Eric Martin and Chris Cundy. 2018. Parallelizing linear recurrent neural nets over sequence length. ICLR. [find](https://scholar.google.com/scholar?q=Eric+Martin+and+Chris+Cundy%2E+2018%2E+Parallelizing+linear+recurrent+neural+nets+over+sequence+length%2E+ICLR%2E)
20. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics.
21. Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. 2014. How to construct deep recurrent neural networks. ICLR. [find](https://scholar.google.com/scholar?q=Razvan+Pascanu%2C+Caglar+Gulcehre%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2014%2E+How+to+construct+deep+recurrent+neural+networks%2E+ICLR%2E)
22. Hoo-Chang Shin, Kirk Roberts, Le Lu, Dina Demner-Fushman, Jianhua Yao, and Ronald M Summers. 2016. Learning to read chest x-rays: recurrent neural cascade model for automated image annotation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2497– 2506. [find](https://scholar.google.com/scholar?q=Hoo%2DChang+Shin%2C+Kirk+Roberts%2C+Le+Lu%2C+Dina+Demner%2DFushman%2C+Jianhua+Yao%2C+and+Ronald+M+Summers%2E+2016%2E+Learning+to+read+chest+x%2Drays%3A+recurrent+neural+cascade+model+for+automated+image+annotation%2E+In+Proc)
23. K. Simonyan and A. Zisserman. 2014. [Very deep convolutional networks for large-scale image recognition](https://md.research.elamur.ai/free/en/papers/simonyan-2014-very-deep-convolutional-networks-large/). CoRR, abs/1409.1556.
24. Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. [Cider: Consensus-based image description evaluation](https://md.research.elamur.ai/free/en/papers/vedantam-2014-cider-consensus-based-image-description/). In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575.
25. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. [Show and tell: A neural image caption generator](https://md.research.elamur.ai/free/en/papers/vinyals-2014-show-tell-neural-image-caption/). In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3156–3164.
26. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. [Show, attend and tell: Neural image caption generation with visual attention](https://md.research.elamur.ai/free/en/papers/xu-2015-show-attend-tell-neural-image/). In International Conference on Machine Learning, pages 2048–2057.

*Updated 24 October 2026*