---
title: "Prose for a Painting — paper by Kashyap et al. (2019)"
description: "It uses an actor-critic design with object, sentiment and scene CNNs, skip-thought poetic… 3.9. BLEU. 13 references, cited by 1 paper here."
authors: ["Prerna Kashyap", "Samrat Phatale", "Iddo Drori"]
year: 2019
venue: "arXiv:1910.03634 · ICCV Workshop on Closing the Loop Between Vision and Language, 2019"
source: "https://arxiv.org/abs/1910.03634"
doi: "10.48550/arxiv.1910.03634"
arxiv: "1910.03634"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kashyap-2019-prose-painting/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kashyap-2019-prose-painting/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/8/) / Prose for a Painting

# Prose for a Painting

**Authors:** Prerna Kashyap, [Samrat Phatale](https://md.research.elamur.ai/free/en/authors/samrat-phatale/) and Iddo Drori · **Year:** 2019 · **Venue:** arXiv:1910.03634 · ICCV Workshop on Closing the Loop Between Vision and Language, 2019 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1910.03634) · [PDF](https://arxiv.org/pdf/1910.03634) · [DOI 10.48550/arxiv.1910.03634](https://doi.org/10.48550/arxiv.1910.03634) · [arXiv 1910.03634](https://arxiv.org/abs/1910.03634) · [OpenAlex](https://openalex.org/W2980223359)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper tackles generating Shakespeare-styled text about a painting. Ordinary image captions are short and literal, and no sizeable corpus pairs paintings with Shakespearean prose exists. The authors therefore chain two separately trained systems. First, an existing image-to-poem model produces a modern English poem. It uses an actor-critic design with object, sentiment and scene CNNs, skip-thought poetic clues, and a policy-gradient generator rewarded by two discriminators. Second, a seq2seq style-transfer model rewrites that poem into Shakespearean English. It is trained on line-aligned modern paraphrases of Shakespeare's plays, with word embeddings retrofitted using a Shakespearean-to-modern dictionary. Attention-based seq2seq with global attention beat a pointer-network variant on BLEU. A small human study rated the outputs moderately well for content, creativity and style. The work matters as a proof of concept: chaining models through an intermediate representation can make up for missing end-to-end parallel data. Code and models are released.

## Key points
- Poem stage data: MultiM-Poem (image–poem pairs), UniM-Poem (large poem corpus), and MultiM-Poem(Ex), which adds each pair's three nearest neighbour poems from the corpus.
- Style-transfer data: modern paraphrases of 16 Shakespeare plays, split into 18,395 training, 1,218 validation and 1,462 test sentences.
- Average target BLEU was 29.65 for seq2seq with global attention, versus 26.97 with a pointer network. Global attention also outperformed Bahdanau attention.
- In the human evaluation, 32 students gave 1–5 Likert ratings: content 3.7, creativity 3.9, Shakespearean style 3.9. BLEU fell as source sentences grew longer.
- Setup: single-layer LSTM encoder and decoder, 192-dimensional retrofitted embeddings shared between source and target, Adam at learning rate 0.001, 25 epochs, TensorFlow 2.0 on Colab.

## Interesting ideas
- When no direct input→target-style data exists, route through an intermediate form that has its own data and keep style as a separate final step; each piece can be trained and checked on its own.
- A small bilingual dictionary (archaic↔modern words) was used to retrofit pretrained embeddings, giving a cheap boost when parallel data is scarce.
- Unknown output tokens are replaced by the source word that received the most attention, a simple copy fallback for names and rare words.
- BLEU is reported as a function of source length rather than as one average, which shows where quality degrades.
- Human ratings were split into separate axes (content relevance, creativity, style fidelity) instead of one overall score.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (back): Generation from the meaning graph could first produce neutral English, then apply a separate register/style pass, so the graph→text step stays explainable and style stays swappable.
- **I2** (tokens): A curated archaic-form table (thou/thee/hath/doth) could feed the lemmatizer and morphology in crate en, so older fable and tale texts map onto modern lemmas.
- **I3** (back): When generating text from the graph, rare names and entities should be copied verbatim from the source mention rather than regenerated, with the alignment logged as the explanation.
- **I4** (learn): Parser and answer evaluations could be bucketed by sentence length so regressions on long sentences are not hidden by the average.
- **I5** (answer): Free-text answers on tales could be judged separately for factual grounding in the text-world and for fluency, keeping correctness failures distinct from wording problems.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Columbia University](https://md.research.elamur.ai/free/en/universities/columbia-university/) — University, New York, United States · 254 works · 56,747 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

## Cite

Cite as: Prerna Kashyap, Samrat Phatale and Iddo Drori. 2019. Prose for a Painting. arXiv:1910.03634 · ICCV Workshop on Closing the Loop Between Vision and Language, 2019.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{kashyap2019prose,
  title = {{Prose for a Painting}},
  author = {Prerna Kashyap, Samrat Phatale and Iddo Drori},
  year = {2019},
  doi = {10.48550/arxiv.1910.03634},
  eprint = {1910.03634},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1910.03634}
}
```

- [← previous Crowdsourced Hedge Term Disambiguation](https://md.research.elamur.ai/free/en/papers/ulinski-2019-crowdsourced-hedge-term-disambiguation/)
- [next → Should a conditional marker arise … The diachronic development of conditional sollte in German](https://md.research.elamur.ai/free/en/papers/breitbarth-2019-i-should-conditional-marker-arise/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/8/)

## References

13 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). International Conference on Learning Representations, 2015. 2 [↗](https://doi.org/10.48550/arxiv.1409.0473)
2. Iddo Drori, Daniel Cohen-Or, and Hezy Yeshurun. Example-based style synthesis. In 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings., 2003. 2 [find](https://scholar.google.com/scholar?q=Iddo+Drori%2C+Daniel+Cohen%2DOr%2C+and+Hezy+Yeshurun%2E+Example%2Dbased+style+synthesis%2E+In+2003+IEEE+Computer+Society+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+2003%2E+Proceedings%2E%2C+2003%2E+2)
3. Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2414–2423, 2016. 2 [find](https://scholar.google.com/scholar?q=Leon+A+Gatys%2C+Alexander+S+Ecker%2C+and+Matthias+Bethge%2E+Image+style+transfer+using+convolutional+neural+networks%2E+In+Proceedings+of+the+IEEE+conference+on+computer+vision+and+pattern+recognition%2C+pages+)
4. Harsh Jhamtani, Varun Gangal, Eduard Hovy, and Eric Nyberg. Shakespearizing modern language using copy-enriched sequence-to-sequence models. ACL Workshop on Stylistic Variation, pages 10–19, 2017. 1, 2 — [Shakespearizing Modern Language Using Copy-Enriched Sequence to Sequence Models](https://md.research.elamur.ai/free/en/papers/jhamtani-2017-shakespearizing-modern-language-copy-enriched/)
5. Prerna Kashyap, Samrat Phatale, and Iddo Drori. Prose for painting: source code. In — [Prose for a Painting](https://md.research.elamur.ai/free/en/papers/kashyap-2019-prose-painting/) [https://github.com/prerna135/prose-for-a-painting](https://github.com/prerna135/prose-for-a-painting), 2019. 4 [↗](https://github.com/prerna135/prose-for-a-painting)
6. Guillaume Lample, Sandeep Subramanian, Eric Smith, Ludovic Denoyer, Y-Lan Boureau, et al. Multiple-attribute text rewriting. International Conference on Learning Representations, 2019. 2 [find](https://scholar.google.com/scholar?q=Guillaume+Lample%2C+Sandeep+Subramanian%2C+Eric+Smith%2C+Ludovic+Denoyer%2C+Y%2DLan+Boureau%2C+et+al%2E+Multiple%2Dattribute+text+rewriting%2E+International+Conference+on+Learning+Representations%2C+2019%2E+2)
7. Bei Liu, Jianlong Fu, Makoto P Kato, and Masatoshi Yoshikawa. Beyond narrative description: generating poetry from images by multi-adversarial training. In ACM International Conference on Multimedia, pages 783–791, 2018. 1, 2 [↗](https://doi.org/10.48550/arxiv.1804.08473)
8. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. [Effective approaches to attention-based neural machine translation](https://md.research.elamur.ai/free/en/papers/luong-2015-effective-approaches-attention-based-neural/). arXiv preprint arXiv:1508.04025, 2015. 2 [↗](https://arxiv.org/abs/1508.04025)
9. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. [Pointer sentinel mixture models](https://md.research.elamur.ai/free/en/papers/merity-2016-pointer-sentinel-mixture-models/). International Conference on Learning Representations, 2017. 2 [↗](https://doi.org/10.48550/arxiv.1609.07843)
10. Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. [Style transfer from non-parallel text by cross-alignment](https://md.research.elamur.ai/free/en/papers/shen-2017-style-transfer-non-parallel-text/). In Advances in neural information processing systems, pages 6830–6841, 2017. 2 [↗](https://doi.org/10.48550/arxiv.1705.09655)
11. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. [Sequence to sequence learning with neural networks](https://md.research.elamur.ai/free/en/papers/sutskever-2014-sequence-sequence-learning-neural-networks/). In Advances in Neural Information Processing Systems, pages 3104–3112, 2014. 1 [↗](https://doi.org/10.48550/arxiv.1409.3215)
12. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. [Show and tell: A neural image caption generator](https://md.research.elamur.ai/free/en/papers/vinyals-2014-show-tell-neural-image-caption/). In Conference on Computer Vision and Pattern Recognition, pages 3156–3164, 2015. 1 [↗](https://doi.org/10.48550/arxiv.1411.4555)
13. Zichao Yang, Zhiting Hu, Chris Dyer, Eric P Xing, and Taylor Berg-Kirkpatrick. [Unsupervised text style transfer using language models as discriminators](https://md.research.elamur.ai/free/en/papers/yang-2018-unsupervised-text-style-transfer-using/). In Advances in Neural Information Processing Systems, pages 7287–7298, 2018. 2 [↗](https://doi.org/10.48550/arxiv.1805.11749)

*Updated 1 November 2026*