---
title: "Reinforcement Learning on Web Interfaces Using"
description: "From each demonstration they build a lattice of \"workflows\": state-independent sequences of action… 52 references, cited by 104 papers here."
authors: ["Evan Zheran Liu", "Kelvin Guu", "Panupong Pasupat", "Tianlin Shi", "Percy Liang"]
year: 2018
venue: "arXiv:1802.08802"
source: "https://arxiv.org/abs/1802.08802"
doi: "10.48550/arxiv.1802.08802"
arxiv: "1802.08802"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liu-2018-reinforcement-learning-web-interfaces-using/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2018-reinforcement-learning-web-interfaces-using/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/) / Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration

# Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration

**Authors:** [Evan Zheran Liu](https://md.research.elamur.ai/free/en/authors/evan-zheran-liu/), [Kelvin Guu](https://md.research.elamur.ai/free/en/authors/kelvin-guu/), [Panupong Pasupat](https://md.research.elamur.ai/free/en/authors/panupong-pasupat/), [Tianlin Shi](https://md.research.elamur.ai/free/en/authors/tianlin-shi/) and [Percy Liang](https://md.research.elamur.ai/free/en/authors/percy-liang/) · **Year:** 2018 · **Venue:** arXiv:1802.08802 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1802.08802) · [PDF](https://arxiv.org/pdf/1802.08802) · [DOI 10.48550/arxiv.1802.08802](https://doi.org/10.48550/arxiv.1802.08802) · [arXiv 1802.08802](https://arxiv.org/abs/1802.08802) · [OpenAlex](https://openalex.org/W2786616516)

**Citations:** 15 citations (OpenAlex, 2026-10-14) · [104](https://md.research.elamur.ai/free/en/papers/liu-2018-reinforcement-learning-web-interfaces-using/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper tackles a core obstacle in teaching RL agents to operate websites (for example, booking a flight or forwarding an email). Rewards come only at the end of an episode and only for fully correct action sequences, so random exploration rarely succeeds. Pre-training on demonstrations through behavioral cloning tends to overfit. The authors instead use a handful of demonstrations to restrict where the agent explores. From each demonstration they build a lattice of "workflows": state-independent sequences of action constraints, such as "click something near a given text, then type a goal field into an input." A simple, environment-blind workflow policy learns via a REINFORCE variant which constraints work. Its successful episodes fill a replay buffer that trains DOMNet, a neural policy that reasons over the HTML DOM tree and the goal. On MiniWoB, the Alaska Airlines booking interface and new harder tasks, the method sets state-of-the-art results using only a few demonstrations. This makes web-operating assistants far more practical to train.

## Key points
- **Framework:** Workflow-guided exploration (WGE) uses demonstrations to prune exploration rather than to imitate them directly. This lets an expressive neural policy (DOMNet) be trained with less risk of overfitting.
- **Constraint language:** Workflow steps are built from a small set of operators:
  - Click and Type actions;
  - element selectors Tag, Text, Like, Near (within 30px), SameRow, SameCol and a Class filter;
  - Field references to the goal.
  
  Nesting is capped at 3 levels.
- **Sample efficiency:** WGE needs only 3–10 demonstrations per task. Earlier MiniWoB work (Shi et al., 2017) used about 200 demonstrations, roughly 10 minutes per task. Compared with behavioral cloning, WGE is over 100x more sample-efficient.
- **Evaluation:** The method is tested on the MiniWoB (World of Bits) benchmark, the Alaska Airlines flight-booking site, and a new task suite. The new suite covers noisy environments, natural-language variation and longer horizons.
- **Setup:** Agents receive +1/−1 reward only at episode end. Goals are given either as key-value structures or as natural-language utterances.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 506 works · 330,053 citations
- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 415 works · 94,028 citations

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works

## Cite

Cite as: Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi and Percy Liang. 2018. Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration. arXiv:1802.08802.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{liu2018reinforcement,
  title = {{Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration}},
  author = {Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi and Percy Liang},
  year = {2018},
  doi = {10.48550/arxiv.1802.08802},
  eprint = {1802.08802},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1802.08802}
}
```

- [← previous Parsing Tweets into Universal Dependencies](https://md.research.elamur.ai/free/en/papers/liu-2018-parsing-tweets-into-universal-dependencies/)
- [next → Sarcasm Analysis Using Conversation Context](https://md.research.elamur.ai/free/en/papers/ghosh-2018-sarcasm-analysis-conversation-context/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/)

## References

52 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. P. Abbeel and A. Ng. Apprenticeship learning via inverse reinforcement learning. In International Conference on Machine Learning (ICML), 2004. [find](https://scholar.google.com/scholar?q=P%2E+Abbeel+and+A%2E+Ng%2E+Apprenticeship+learning+via+inverse+reinforcement+learning%2E+In+International+Conference+on+Machine+Learning+%28ICML%29%2C+2004%2E)
2. J. Allen, N. Chambers, G. Ferguson, L. Galescu, H. Jung, M. Swift, and W. Taysom. PLOW: A collaborative task learning agent. In Association for the Advancement of Artificial Intelligence (AAAI), pp. 1514–1519, 2007. [find](https://scholar.google.com/scholar?q=J%2E+Allen%2C+N%2E+Chambers%2C+G%2E+Ferguson%2C+L%2E+Galescu%2C+H%2E+Jung%2C+M%2E+Swift%2C+and+W%2E+Taysom%2E+PLOW%3A+A+collaborative+task+learning+agent%2E+In+Association+for+the+Advancement+of+Artificial+Intelligence+%28AAAI%29%2C+pp%2E+1)
3. D. Andre. Programmable reinforcement learning agents. PhD thesis, University of California, Berkeley, 2003. [find](https://scholar.google.com/scholar?q=D%2E+Andre%2E+Programmable+reinforcement+learning+agents%2E+PhD+thesis%2C+University+of+California%2C+Berkeley%2C+2003%2E)
4. J. Andreas, D. Klein, and S. Levine. Modular multitask reinforcement learning with policy sketches. arXiv preprint arXiv:1611.01796, 2016. [find](https://scholar.google.com/scholar?q=J%2E+Andreas%2C+D%2E+Klein%2C+and+S%2E+Levine%2E+Modular+multitask+reinforcement+learning+with+policy+sketches%2E+arXiv+preprint+arXiv%3A1611%2E01796%2C+2016%2E)
5. M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba. [Hindsight experience replay](https://md.research.elamur.ai/free/en/papers/andrychowicz-2017-hindsight-experience-replay/). arXiv preprint arXiv:1707.01495, 2017.
6. C. G. Atkeson and S. Schaal. Robot learning from demonstration. In International Conference on Machine Learning (ICML), volume 97, pp. 12–20, 1997. [find](https://scholar.google.com/scholar?q=C%2E+G%2E+Atkeson+and+S%2E+Schaal%2E+Robot+learning+from+demonstration%2E+In+International+Conference+on+Machine+Learning+%28ICML%29%2C+volume+97%2C+pp%2E+12%E2%80%9320%2C+1997%2E)
7. P. Bacon, J. Harb, and D. Precup. The option-critic architecture. In Association for the Advancement of Artificial Intelligence (AAAI), pp. 1726–1734, 2017. [find](https://scholar.google.com/scholar?q=P%2E+Bacon%2C+J%2E+Harb%2C+and+D%2E+Precup%2E+The+option%2Dcritic+architecture%2E+In+Association+for+the+Advancement+of+Artificial+Intelligence+%28AAAI%29%2C+pp%2E+1726%E2%80%931734%2C+2017%2E)
8. N. Baram, O. Anschel, I. Caspi, and S. Mannor. End-to-end differentiable adversarial imitation learning. In International Conference on Machine Learning (ICML), pp. 390–399, 2017. [find](https://scholar.google.com/scholar?q=N%2E+Baram%2C+O%2E+Anschel%2C+I%2E+Caspi%2C+and+S%2E+Mannor%2E+End%2Dto%2Dend+differentiable+adversarial+imitation+learning%2E+In+International+Conference+on+Machine+Learning+%28ICML%29%2C+pp%2E+390%E2%80%93399%2C+2017%2E)
9. M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. [The arcade learning environment: An evaluation platform for general agents](https://md.research.elamur.ai/free/en/papers/bellemare-2012-arcade-learning-environment-evaluation-platform/). Journal of Artificial Intelligence Research (JAIR), 47: 253–279, 2013.
10. S. Bitzer, M. Howard, and S. Vijayakumar. Using dimensionality reduction to exploit constraints in reinforcement learning. In International Conference on Intelligent Robots and Systems (IROS), pp. 3219–3225, 2010. [find](https://scholar.google.com/scholar?q=S%2E+Bitzer%2C+M%2E+Howard%2C+and+S%2E+Vijayakumar%2E+Using+dimensionality+reduction+to+exploit+constraints+in+reinforcement+learning%2E+In+International+Conference+on+Intelligent+Robots+and+Systems+%28IROS%29%2C+pp%2E+321)
11. T. Brys, A. Harutyunyan, H. B. Suay, S. Chernova, M. E. Taylor, and A. Now’e. Reinforcement learning from demonstration through shaping. In International Joint Conference on Artificial Intelligence (IJCAI), 2015. [find](https://scholar.google.com/scholar?q=T%2E+Brys%2C+A%2E+Harutyunyan%2C+H%2E+B%2E+Suay%2C+S%2E+Chernova%2C+M%2E+E%2E+Taylor%2C+and+A%2E+Now%E2%80%99e%2E+Reinforcement+learning+from+demonstration+through+shaping%2E+In+International+Joint+Conference+on+Artificial+Intelligence+%28I)
12. N. Chentanez, A. G. Barto, and S. P. Singh. Intrinsically motivated reinforcement learning. In Advances in Neural Information Processing Systems (NIPS), pp. 1281–1288, 2005. [find](https://scholar.google.com/scholar?q=N%2E+Chentanez%2C+A%2E+G%2E+Barto%2C+and+S%2E+P%2E+Singh%2E+Intrinsically+motivated+reinforcement+learning%2E+In+Advances+in+Neural+Information+Processing+Systems+%28NIPS%29%2C+pp%2E+1281%E2%80%931288%2C+2005%2E)
13. C. Daniel, G. Neumann, O. Kroemer, and J. Peters. Hierarchical relative entropy policy search. Journal of Machine Learning Research (JMLR), 17:3190–3239, 2016. [find](https://scholar.google.com/scholar?q=C%2E+Daniel%2C+G%2E+Neumann%2C+O%2E+Kroemer%2C+and+J%2E+Peters%2E+Hierarchical+relative+entropy+policy+search%2E+Journal+of+Machine+Learning+Research+%28JMLR%29%2C+17%3A3190%E2%80%933239%2C+2016%2E)
14. B. Deka, Z. Huang, and R. Kumar. Erica: Interaction mining mobile apps. In User Interface Software and Technology (UIST), pp. 767–776, 2016. [find](https://scholar.google.com/scholar?q=B%2E+Deka%2C+Z%2E+Huang%2C+and+R%2E+Kumar%2E+Erica%3A+Interaction+mining+mobile+apps%2E+In+User+Interface+Software+and+Technology+%28UIST%29%2C+pp%2E+767%E2%80%93776%2C+2016%2E)
15. T. G. Dietterich. The MAXQ method for hierarchical reinforcement learning. In International Conference on Machine Learning (ICML), 1998. [find](https://scholar.google.com/scholar?q=T%2E+G%2E+Dietterich%2E+The+MAXQ+method+for+hierarchical+reinforcement+learning%2E+In+International+Conference+on+Machine+Learning+%28ICML%29%2C+1998%2E)
16. C. Finn, S. Levine, and P. Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference on Machine Learning (ICML), pp. 49–58, 2016. [find](https://scholar.google.com/scholar?q=C%2E+Finn%2C+S%2E+Levine%2C+and+P%2E+Abbeel%2E+Guided+cost+learning%3A+Deep+inverse+optimal+control+via+policy+optimization%2E+In+International+Conference+on+Machine+Learning+%28ICML%29%2C+pp%2E+49%E2%80%9358%2C+2016%2E)
17. W. L. Hamilton, R. Ying, and J. Leskovec. [Inductive representation learning on large graphs](https://md.research.elamur.ai/free/en/papers/hamilton-2017-inductive-representation-learning-large-graphs/). In Advances in Neural Information Processing Systems (NIPS), 2017.
18. K. Hauser, T. Bretl, K. Harada, and J. Latombe. Using motion primitives in probabilistic sample-based planning for humanoid robots. Algorithmic foundation of robotics, 7:507–522, 2008. [find](https://scholar.google.com/scholar?q=K%2E+Hauser%2C+T%2E+Bretl%2C+K%2E+Harada%2C+and+J%2E+Latombe%2E+Using+motion+primitives+in+probabilistic+sample%2Dbased+planning+for+humanoid+robots%2E+Algorithmic+foundation+of+robotics%2C+7%3A507%E2%80%93522%2C+2008%2E)
19. T. Hester, M. Vecerik, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, A. Sendonaris, G. Dulac-Arnold, I. Osband, J. Agapiou, J. Z. Leibo, and A. Gruslys. Deep Q-learning from demonstrations. In Association for the Advancement of Artificial Intelligence (AAAI), 2018. [find](https://scholar.google.com/scholar?q=T%2E+Hester%2C+M%2E+Vecerik%2C+O%2E+Pietquin%2C+M%2E+Lanctot%2C+T%2E+Schaul%2C+B%2E+Piot%2C+A%2E+Sendonaris%2C+G%2E+Dulac%2DArnold%2C+I%2E+Osband%2C+J%2E+Agapiou%2C+J%2E+Z%2E+Leibo%2C+and+A%2E+Gruslys%2E+Deep+Q%2Dlearning+from+demonstrations%2E+In+Associat)
20. J. Ho and S. Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems (NIPS), pp. 4565–4573, 2016. [find](https://scholar.google.com/scholar?q=J%2E+Ho+and+S%2E+Ermon%2E+Generative+adversarial+imitation+learning%2E+In+Advances+in+Neural+Information+Processing+Systems+%28NIPS%29%2C+pp%2E+4565%E2%80%934573%2C+2016%2E)
21. I. Hosu and T. Rebedea. Playing Atari games with deep reinforcement learning and human checkpoint replay. In Evaluating General Purpose AI, 2016. [find](https://scholar.google.com/scholar?q=I%2E+Hosu+and+T%2E+Rebedea%2E+Playing+Atari+games+with+deep+reinforcement+learning+and+human+checkpoint+replay%2E+In+Evaluating+General+Purpose+AI%2C+2016%2E)
22. A. Hussein, E. Elyan, M. M. Gaber, and C. Jayne. Deep reward shaping from demonstrations. In International Joint Conference on Neural Networks, 2017. [find](https://scholar.google.com/scholar?q=A%2E+Hussein%2C+E%2E+Elyan%2C+M%2E+M%2E+Gaber%2C+and+C%2E+Jayne%2E+Deep+reward+shaping+from+demonstrations%2E+In+International+Joint+Conference+on+Neural+Networks%2C+2017%2E)
23. M. Jaderberg, V. Mnih, W. M. Czarnecki, T. Schaul, J. Z. Leibo, D. Silver, and K. Kavukcuoglu. [Reinforcement learning with unsupervised auxiliary tasks](https://md.research.elamur.ai/free/en/papers/jaderberg-2016-reinforcement-learning-unsupervised-auxiliary-tasks/). arXiv preprint arXiv:1611.05397, 2016.
24. B. Kim, A. massoud Farahmand, J. Pineau, and D. Precup. Learning from limited demonstrations. In Advances in Neural Information Processing Systems (NIPS), pp. 2859–2867, 2013. [find](https://scholar.google.com/scholar?q=B%2E+Kim%2C+A%2E+massoud+Farahmand%2C+J%2E+Pineau%2C+and+D%2E+Precup%2E+Learning+from+limited+demonstrations%2E+In+Advances+in+Neural+Information+Processing+Systems+%28NIPS%29%2C+pp%2E+2859%E2%80%932867%2C+2013%2E)
25. T. N. Kipf and M. Welling. [Semi-supervised classification with graph convolutional networks](https://md.research.elamur.ai/free/en/papers/kipf-2016-semi-supervised-classification-graph-convolutional/). In International Conference on Learning Representations (ICLR), 2017.
26. G. Konidaris and A. G. Barto. Building portable options: Skill transfer in reinforcement learning. In International Joint Conference on Artificial Intelligence (IJCAI), 2007. [find](https://scholar.google.com/scholar?q=G%2E+Konidaris+and+A%2E+G%2E+Barto%2E+Building+portable+options%3A+Skill+transfer+in+reinforcement+learning%2E+In+International+Joint+Conference+on+Artificial+Intelligence+%28IJCAI%29%2C+2007%2E)
27. S. Levine and V. Koltun. Guided policy search. In International Conference on Machine Learning (ICML), 2013. [find](https://scholar.google.com/scholar?q=S%2E+Levine+and+V%2E+Koltun%2E+Guided+policy+search%2E+In+International+Conference+on+Machine+Learning+%28ICML%29%2C+2013%2E)
28. L. Li, T. J. Walsh, and M. L. Littman. Towards a unified theory of state abstraction for mdps. In International Symposium on Artificial Intelligence and Mathematics (ISAIM), 2006. [find](https://scholar.google.com/scholar?q=L%2E+Li%2C+T%2E+J%2E+Walsh%2C+and+M%2E+L%2E+Littman%2E+Towards+a+unified+theory+of+state+abstraction+for+mdps%2E+In+International+Symposium+on+Artificial+Intelligence+and+Mathematics+%28ISAIM%29%2C+2006%2E)
29. B. Marthi and C. Guestrin. Concurrent hierarchical reinforcement learning. In International Joint Conference on Artificial Intelligence (IJCAI), 2005. [find](https://scholar.google.com/scholar?q=B%2E+Marthi+and+C%2E+Guestrin%2E+Concurrent+hierarchical+reinforcement+learning%2E+In+International+Joint+Conference+on+Artificial+Intelligence+%28IJCAI%29%2C+2005%2E)
30. V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. P. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning (ICML), 2016. [find](https://scholar.google.com/scholar?q=V%2E+Mnih%2C+A%2E+P%2E+Badia%2C+M%2E+Mirza%2C+A%2E+Graves%2C+T%2E+P%2E+Lillicrap%2C+T%2E+Harley%2C+D%2E+Silver%2C+and+K%2E+Kavukcuoglu%2E+Asynchronous+methods+for+deep+reinforcement+learning%2E+In+International+Conference+on+Machine+Learn)
31. A. Nair, B. McGrew, M. Andrychowicz, W. Zaremba, and P. Abbeel. Overcoming exploration in reinforcement learning with demonstrations. arXiv preprint arXiv:1709.10089, 2017. [find](https://scholar.google.com/scholar?q=A%2E+Nair%2C+B%2E+McGrew%2C+M%2E+Andrychowicz%2C+W%2E+Zaremba%2C+and+P%2E+Abbeel%2E+Overcoming+exploration+in+reinforcement+learning+with+demonstrations%2E+arXiv+preprint+arXiv%3A1709%2E10089%2C+2017%2E)
32. A. Y. Ng, D. Harada, and S. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In International Conference on Machine Learning (ICML), volume 99, pp. 278–287, 1999. [find](https://scholar.google.com/scholar?q=A%2E+Y%2E+Ng%2C+D%2E+Harada%2C+and+S%2E+Russell%2E+Policy+invariance+under+reward+transformations%3A+Theory+and+application+to+reward+shaping%2E+In+International+Conference+on+Machine+Learning+%28ICML%29%2C+volume+99%2C+pp%2E+27)
33. I. Osband, C. Blundell, A. Pritzel, and B. V. Roy. [Deep exploration via bootstrapped DQN](https://md.research.elamur.ai/free/en/papers/osband-2016-deep-exploration-bootstrapped-dqn/). In Advances In Neural Information Processing Systems, pp. 4026–4034, 2016.
34. R. Parr and S. J. Russell. Reinforcement learning with hierarchies of machines. In Advances in Neural Information Processing Systems (NIPS), pp. 1043–1049, 1998. [find](https://scholar.google.com/scholar?q=R%2E+Parr+and+S%2E+J%2E+Russell%2E+Reinforcement+learning+with+hierarchies+of+machines%2E+In+Advances+in+Neural+Information+Processing+Systems+%28NIPS%29%2C+pp%2E+1043%E2%80%931049%2C+1998%2E)
35. C. Perez-D’Arpino and J. A. Shah. C-learn: Learning geometric constraints from demonstrations for multi-step manipulation in shared autonomy. In International Conference on Robotics and Automation (ICRA), pp. 4058–4065, 2017. [find](https://scholar.google.com/scholar?q=C%2E+Perez%2DD%E2%80%99Arpino+and+J%2E+A%2E+Shah%2E+C%2Dlearn%3A+Learning+geometric+constraints+from+demonstrations+for+multi%2Dstep+manipulation+in+shared+autonomy%2E+In+International+Conference+on+Robotics+and+Automation+%28IC)
36. T. Pham, T. Tran, D. Phung, and S. Venkatesh. Column networks for collective classification. In Association for the Advancement of Artificial Intelligence (AAAI), 2017. [find](https://scholar.google.com/scholar?q=T%2E+Pham%2C+T%2E+Tran%2C+D%2E+Phung%2C+and+S%2E+Venkatesh%2E+Column+networks+for+collective+classification%2E+In+Association+for+the+Advancement+of+Artificial+Intelligence+%28AAAI%29%2C+2017%2E)
37. M. Phillips, V. Hwang, S. Chitta, and M. Likhachev. Learning to plan for constrained manipulation from demonstrations. Autonomous Robots, 40(1):109–124, 2016. [find](https://scholar.google.com/scholar?q=M%2E+Phillips%2C+V%2E+Hwang%2C+S%2E+Chitta%2C+and+M%2E+Likhachev%2E+Learning+to+plan+for+constrained+manipulation+from+demonstrations%2E+Autonomous+Robots%2C+40%281%29%3A109%E2%80%93124%2C+2016%2E)
38. D. A. Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural Computation, 3(1):88–97, 1991. [find](https://scholar.google.com/scholar?q=D%2E+A%2E+Pomerleau%2E+Efficient+training+of+artificial+neural+networks+for+autonomous+navigation%2E+Neural+Computation%2C+3%281%29%3A88%E2%80%9397%2C+1991%2E)
39. M. Roderick, C. Grimm, and S. Tellex. Deep abstract Q-networks. arXiv preprint arXiv:1710.00459, 2017. — [arXiv:1710.00459](https://arxiv.org/abs/1710.00459) [find](https://scholar.google.com/scholar?q=M%2E+Roderick%2C+C%2E+Grimm%2C+and+S%2E+Tellex%2E+Deep+abstract+Q%2Dnetworks%2E+arXiv+preprint+arXiv%3A1710%2E00459%2C+2017%2E+%E2%80%94+%5BarXiv%3A1710%2E00459%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1710%2E00459%29)
40. S. Ross and J. A. Bagnell. Reinforcement and imitation learning via interactive no-regret learning. arXiv preprint arXiv:1406.5979, 2014. [find](https://scholar.google.com/scholar?q=S%2E+Ross+and+J%2E+A%2E+Bagnell%2E+Reinforcement+and+imitation+learning+via+interactive+no%2Dregret+learning%2E+arXiv+preprint+arXiv%3A1406%2E5979%2C+2014%2E)
41. S. Ross, G. Gordon, and A. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Artificial Intelligence and Statistics (AISTATS), 2011. [find](https://scholar.google.com/scholar?q=S%2E+Ross%2C+G%2E+Gordon%2C+and+A%2E+Bagnell%2E+A+reduction+of+imitation+learning+and+structured+prediction+to+no%2Dregret+online+learning%2E+In+Artificial+Intelligence+and+Statistics+%28AISTATS%29%2C+2011%2E)
42. T. Shi, A. Karpathy, L. Fan, J. Hernandez, and P. Liang. World of bits: An open-domain platform for web-based agents. In International Conference on Machine Learning (ICML), 2017. [find](https://scholar.google.com/scholar?q=T%2E+Shi%2C+A%2E+Karpathy%2C+L%2E+Fan%2C+J%2E+Hernandez%2C+and+P%2E+Liang%2E+World+of+bits%3A+An+open%2Ddomain+platform+for+web%2Dbased+agents%2E+In+International+Conference+on+Machine+Learning+%28ICML%29%2C+2017%2E)
43. B. Stone and S. Soper. Amazon Unveils a Listening, Talking, Music-Playing Speaker for Your Home. Bloomberg L. P., 2014. [find](https://scholar.google.com/scholar?q=B%2E+Stone+and+S%2E+Soper%2E+Amazon+Unveils+a+Listening%2C+Talking%2C+Music%2DPlaying+Speaker+for+Your+Home%2E+Bloomberg+L%2E+P%2E%2C+2014%2E)
44. W. Sun, A. Venkatraman, G. J. Gordon, B. Boots, and J. A. Bagnell. Deeply aggrevated: Differentiable imitation learning for sequential prediction. In International Conference on Machine Learning (ICML), 2017. [find](https://scholar.google.com/scholar?q=W%2E+Sun%2C+A%2E+Venkatraman%2C+G%2E+J%2E+Gordon%2C+B%2E+Boots%2C+and+J%2E+A%2E+Bagnell%2E+Deeply+aggrevated%3A+Differentiable+imitation+learning+for+sequential+prediction%2E+In+International+Conference+on+Machine+Learning+%28ICML)
45. R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction, volume 1. MIT Press MIT press Cambridge, 1998. [find](https://scholar.google.com/scholar?q=R%2E+S%2E+Sutton+and+A%2E+G%2E+Barto%2E+Reinforcement+learning%3A+An+introduction%2C+volume+1%2E+MIT+Press+MIT+press+Cambridge%2C+1998%2E)
46. R. S. Sutton, D. Precup, and S. Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Articial intelligence, 112:181–211, 1999. [find](https://scholar.google.com/scholar?q=R%2E+S%2E+Sutton%2C+D%2E+Precup%2C+and+S%2E+Singh%2E+Between+mdps+and+semi%2Dmdps%3A+A+framework+for+temporal+abstraction+in+reinforcement+learning%2E+Articial+intelligence%2C+112%3A181%E2%80%93211%2C+1999%2E)
47. M. Vecerik, T. Hester, J. Scholz, F. Wang, O. Pietquin, B. Piot, N. Heess, T. Rothorl, T. Lampe, and M. Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv:1707.08817, 2017. — [arXiv:1707.08817](https://arxiv.org/abs/1707.08817) [find](https://scholar.google.com/scholar?q=M%2E+Vecerik%2C+T%2E+Hester%2C+J%2E+Scholz%2C+F%2E+Wang%2C+O%2E+Pietquin%2C+B%2E+Piot%2C+N%2E+Heess%2C+T%2E+Rothorl%2C+T%2E+Lampe%2C+and+M%2E+Riedmiller%2E+Leveraging+demonstrations+for+deep+reinforcement+learning+on+robotics+problems+with+)
48. Y. Wang, J. Berant, and P. Liang. Building a semantic parser overnight. In Association for Computational Linguistics (ACL), 2015. [find](https://scholar.google.com/scholar?q=Y%2E+Wang%2C+J%2E+Berant%2C+and+P%2E+Liang%2E+Building+a+semantic+parser+overnight%2E+In+Association+for+Computational+Linguistics+%28ACL%29%2C+2015%2E)
49. T. Weber, S. Racanière, D. P. Reichert, L. Buesing, A. Guez, D. J. Rezende, A. P. Badia, O. Vinyals, N. Heess, Y. Li, et al. Imagination-augmented agents for deep reinforcement learning. arXiv preprint arXiv:1707.06203, 2017. [find](https://scholar.google.com/scholar?q=T%2E+Weber%2C+S%2E+Racani%C3%A8re%2C+D%2E+P%2E+Reichert%2C+L%2E+Buesing%2C+A%2E+Guez%2C+D%2E+J%2E+Rezende%2C+A%2E+P%2E+Badia%2C+O%2E+Vinyals%2C+N%2E+Heess%2C+Y%2E+Li%2C+et+al%2E+Imagination%2Daugmented+agents+for+deep+reinforcement+learning%2E+arXiv+preprin)
50. R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229–256, 1992. [find](https://scholar.google.com/scholar?q=R%2E+J%2E+Williams%2E+Simple+statistical+gradient%2Dfollowing+algorithms+for+connectionist+reinforcement+learning%2E+Machine+learning%2C+8%283%29%3A229%E2%80%93256%2C+1992%2E)
51. T. Yeh, T. Chang, and R. Miller. Sikuli: using GUI screenshots for search and automation. In User Interface Software and Technology (UIST), 2009. [find](https://scholar.google.com/scholar?q=T%2E+Yeh%2C+T%2E+Chang%2C+and+R%2E+Miller%2E+Sikuli%3A+using+GUI+screenshots+for+search+and+automation%2E+In+User+Interface+Software+and+Technology+%28UIST%29%2C+2009%2E)
52. B. D. Ziebart, A. L. Maas, J. A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforcement learning. In Association for the Advancement of Artificial Intelligence (AAAI), 2008. [find](https://scholar.google.com/scholar?q=B%2E+D%2E+Ziebart%2C+A%2E+L%2E+Maas%2C+J%2E+A%2E+Bagnell%2C+and+A%2E+K%2E+Dey%2E+Maximum+entropy+inverse+reinforcement+learning%2E+In+Association+for+the+Advancement+of+Artificial+Intelligence+%28AAAI%29%2C+2008%2E)

*Updated 2 November 2026*