---
title: "Advantage-Weighted Regression — paper by Peng et al. (2019)"
description: "The authors propose advantage-weighted regression (AWR), an off-policy, model-free algorithm built… 39 references, cited by 65 papers here."
authors: ["Xue Bin Peng", "Aviral Kumar", "Grace Zhang", "Sergey Levine"]
year: 2019
venue: "arXiv:1910.00177"
source: "https://arxiv.org/abs/1910.00177"
doi: "10.48550/arxiv.1910.00177"
arxiv: "1910.00177"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/peng-2019-advantage-weighted-regression-simple-scalable/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/peng-2019-advantage-weighted-regression-simple-scalable/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) / Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning

# Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning

**Authors:** Xue Bin Peng, [Aviral Kumar](https://md.research.elamur.ai/free/en/authors/aviral-kumar/), Grace Zhang and [Sergey Levine](https://md.research.elamur.ai/free/en/authors/sergey-levine/) · **Year:** 2019 · **Venue:** arXiv:1910.00177 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1910.00177) · [PDF](https://arxiv.org/pdf/1910.00177) · [DOI 10.48550/arxiv.1910.00177](https://doi.org/10.48550/arxiv.1910.00177) · [arXiv 1910.00177](https://arxiv.org/abs/1910.00177) · [OpenAlex](https://openalex.org/W2978455699)

**Citations:** 158 citations (OpenAlex, 2026-10-14) · [65](https://md.research.elamur.ai/free/en/papers/peng-2019-advantage-weighted-regression-simple-scalable/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Reinforcement learning algorithms are often harder to implement and tune than supervised learning. Policy gradient methods are simple but tend to be unstable and largely on-policy, so they waste samples. The authors propose advantage-weighted regression (AWR), an off-policy, model-free algorithm built entirely from supervised learning steps. In each iteration, the agent adds fresh trajectories to a FIFO replay buffer. A value function is then fit to return estimates from that buffer. Finally, the policy is trained by maximum likelihood on buffered actions, with each action weighted by its exponentiated advantage.

AWR revisits reward-weighted regression (RWR). The main change is that it optimizes expected improvement rather than expected return, which introduces a value baseline into the weights. The authors derive AWR as an approximate solution to a KL-constrained policy search problem and extend the analysis to replay data drawn from a mixture of past policies. On OpenAI Gym benchmarks, AWR is competitive with established on- and off-policy methods. It also learns well from static datasets and handles complex simulated characters. Its significance lies in combining a few lines of code with off-policy data reuse.

## Key points
- Each AWR iteration has two regression steps: fitting a value function, then fitting the policy with weights exp(advantage/β). It supports both continuous and discrete actions.
- The advantage baseline is the key difference from RWR (Peters & Schaal, 2007) and yields large empirical gains. AWR is also much simpler than MPO.
- With experience replay, a single mean value function approximates the weighted average of past policies' values. The resulting single-rollout estimator is biased but works in practice.
- The experimental setup uses:
  - policy and value networks with hidden layers of 128 and 64 ReLU units, trained by SGD with momentum 0.9;
  - β = 0.05, TD(λ) with λ = 0.95, and a weight clip of 20;
  - about 2000 samples collected per iteration, a 50k-sample buffer, and minibatches of 256;
  - 200 value updates and 1000 policy updates per iteration.
- On Gym tasks, results are averaged over 5 seeds. On motion imitation tasks (e.g., a Humanoid spinkick and a Dog canter), AWR typically reaches good performance more quickly than either PPO or RWR.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 402 works · 212,763 citations
- [University of California, Berkeley](https://md.research.elamur.ai/free/en/universities/university-of-california-berkeley/) — University, Berkeley, United States · 209 works · 244,782 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🧮 Computational Theory and Mathematics](https://md.research.elamur.ai/free/en/shelves/computer-science/computational-theory-and-mathematics/) · 321 works · [🎯 Decision Sciences](https://md.research.elamur.ai/free/en/shelves/decision-sciences/) · 249 works · [🎯 Management Science and Operations Research](https://md.research.elamur.ai/free/en/shelves/decision-sciences/management-science-and-operations-research/) · 134 works

## Cite

Cite as: Xue Bin Peng, Aviral Kumar, Grace Zhang and Sergey Levine. 2019. Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning. arXiv:1910.00177.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{peng2019advantage,
  title = {{Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning}},
  author = {Xue {Bin Peng}, Aviral Kumar, Grace Zhang and Sergey Levine},
  year = {2019},
  doi = {10.48550/arxiv.1910.00177},
  eprint = {1910.00177},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1910.00177}
}
```

- [← previous A Hierarchically-Labeled Portuguese Hate Speech Dataset](https://md.research.elamur.ai/free/en/papers/fortuna-2019-hierarchically-labeled-portuguese-hate-speech/)
- [next → Beyond BLEU: Training Neural Machine Translation with Semantic Similarity](https://md.research.elamur.ai/free/en/papers/wieting-2019-beyond-bleu-training-neural-machine/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/)

## References

39 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. In International Conference on Learning Representations, 2018. URL [https://openreview.net/forum?id=S1ANxQW0b](https://openreview.net/forum?id=S1ANxQW0b). [↗](https://openreview.net/forum?id=S1ANxQW0b)
2. Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016. [find](https://scholar.google.com/scholar?q=Greg+Brockman%2C+Vicki+Cheung%2C+Ludwig+Pettersson%2C+Jonas+Schneider%2C+John+Schulman%2C+Jie+Tang%2C+and+Wojciech+Zaremba%2E+Openai+gym%2C+2016%2E)
3. CMU. Cmu graphics lab motion capture database. [http://mocap.cs.cmu.edu/](http://mocap.cs.cmu.edu/). Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https://github.com/openai/baselines, 2017. [↗](http://mocap.cs.cmu.edu/)
4. Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In ICML, pp. 1329–1338, 2016. URL [http://proceedings.mlr.press/v48/duan16.html](http://proceedings.mlr.press/v48/duan16.html). [↗](http://proceedings.mlr.press/v48/duan16.html)
5. Justin Fu, Aviral Kumar, Matthew Soh, and Sergey Levine. Diagnosing bottlenecks in deep q-learning algorithms. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2021–… [find](https://scholar.google.com/scholar?q=Justin+Fu%2C+Aviral+Kumar%2C+Matthew+Soh%2C+and+Sergey+Levine%2E+Diagnosing+bottlenecks+in+deep+q%2Dlearning+algorithms%2E+In+Kamalika+Chaudhuri+and+Ruslan+Salakhutdinov+%28eds%2E%29%2C+Proceedings+of+the+36th+Internatio)
6. Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1587–1596, Stoc… [find](https://scholar.google.com/scholar?q=Scott+Fujimoto%2C+Herke+van+Hoof%2C+and+David+Meger%2E+Addressing+function+approximation+error+in+actor%2Dcritic+methods%2E+In+Jennifer+Dy+and+Andreas+Krause+%28eds%2E%29%2C+Proceedings+of+the+35th+International+Confer)
7. Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2052–2062… [find](https://scholar.google.com/scholar?q=Scott+Fujimoto%2C+David+Meger%2C+and+Doina+Precup%2E+Off%2Dpolicy+deep+reinforcement+learning+without+exploration%2E+In+Kamalika+Chaudhuri+and+Ruslan+Salakhutdinov+%28eds%2E%29%2C+Proceedings+of+the+36th+International+)
8. Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous deep q-learning with model-based acceleration. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Res… [find](https://scholar.google.com/scholar?q=Shixiang+Gu%2C+Timothy+Lillicrap%2C+Ilya+Sutskever%2C+and+Sergey+Levine%2E+Continuous+deep+q%2Dlearning+with+model%2Dbased+acceleration%2E+In+Maria+Florina+Balcan+and+Kilian+Q%2E+Weinberger+%28eds%2E%29%2C+Proceedings+of+The)
9. Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings… [find](https://scholar.google.com/scholar?q=Tuomas+Haarnoja%2C+Aurick+Zhou%2C+Pieter+Abbeel%2C+and+Sergey+Levine%2E+Soft+actor%2Dcritic%3A+Off%2Dpolicy+maximum+entropy+deep+reinforcement+learning+with+a+stochastic+actor%2E+In+Jennifer+Dy+and+Andreas+Krause+%28ed)
10. Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings… [find](https://scholar.google.com/scholar?q=Tuomas+Haarnoja%2C+Aurick+Zhou%2C+Pieter+Abbeel%2C+and+Sergey+Levine%2E+Soft+actor%2Dcritic%3A+Off%2Dpolicy+maximum+entropy+deep+reinforcement+learning+with+a+stochastic+actor%2E+In+Jennifer+Dy+and+Andreas+Krause+%28ed)
11. Hirotaka Hachiya, Takayuki Akiyama, Masashi Sugiayma, and Jan Peters. Adaptive importance sampling for value function approximation in off-policy reinforcement learning. Neural Netw., 22(10):1399–1410, December 2009. ISSN 0893-6080. doi: 10.1016/j.neunet.2009.01.002. URL http://dx.doi.org/10.1016/j.… — [doi:10.1016/j.neunet.2009.01.002](https://doi.org/10.1016/j.neunet.2009.01.002) [↗](http://dx.doi.org/10.1016/j.…)
12. Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI’16, pp. 2094–2100. AAAI Press, 2016. URL [http://dl.acm.org/citation.cfm?id=3016100.3016191](http://dl.acm.org/citation.cfm?id=3016100.3016191). [↗](http://dl.acm.org/citation.cfm?id=3016100.3016191)
13. Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, S. M. Ali Eslami, Martin A. Riedmiller, and David Silver. Emergence of locomotion behaviours in rich environments. CoRR, abs/1707.02286, 2017. URL http://arxiv.org/abs/1707.02286. — [arXiv:1707.02286](https://arxiv.org/abs/1707.02286) [↗](http://arxiv.org/abs/1707.02286)
14. Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Daniel Horgan, Bilal Piot, Mohammad Gheshlaghi Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. CoRR, abs/1710.02298, 2017. URL http://arxiv.org/abs/1710.02298. [↗](http://arxiv.org/abs/1710.02298)
15. Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning, ICML ’02, pp. 267–274, San Francisco, CA, USA, 2002. Morgan Kaufmann Publishers Inc. ISBN 1-55860-873-7. URL [http://dl.acm.org/citat…](http://dl.acm.org/citat…) [↗](http://dl.acm.org/citat…)
16. Jens Kober and Jan R. Peters. Policy search for motor primitives in robotics. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou (eds.), Advances in Neural Information Processing Systems 21, pp. 849–856. Curran Associates, Inc., 2009. URL [http://papers.nips.cc/paper/3545-policy-search-for-motor-p…](http://papers.nips.cc/paper/3545-policy-search-for-motor-p…) [↗](http://papers.nips.cc/paper/3545-policy-search-for-motor-p…)
17. Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. CoRR, abs/1906.00949, 2019. URL http://arxiv.org/abs/1906.00949. — [arXiv:1906.00949](https://arxiv.org/abs/1906.00949) [↗](http://arxiv.org/abs/1906.00949)
18. Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Manfred Otto Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. ICLR, 2016. [find](https://scholar.google.com/scholar?q=Timothy+P%2E+Lillicrap%2C+Jonathan+J%2E+Hunt%2C+Alexander+Pritzel%2C+Nicolas+Manfred+Otto+Heess%2C+Tom+Erez%2C+Yuval+Tassa%2C+David+Silver%2C+and+Daan+Wierstra%2E+Continuous+control+with+deep+reinforcement+learning%2E+ICLR)
19. Nicolas Meuleau, Leonid Peshkin, Leslie P. Kaelbling, and Kee eung Kim. Off-policy policy search. Technical report, 2000. [find](https://scholar.google.com/scholar?q=Nicolas+Meuleau%2C+Leonid+Peshkin%2C+Leslie+P%2E+Kaelbling%2C+and+Kee+eung+Kim%2E+Off%2Dpolicy+policy+search%2E+Technical+report%2C+2000%2E)
20. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis H… [find](https://scholar.google.com/scholar?q=Volodymyr+Mnih%2C+Koray+Kavukcuoglu%2C+David+Silver%2C+Andrei+A%2E+Rusu%2C+Joel+Veness%2C+Marc+G%2E+Bellemare%2C+Alex+Graves%2C+Martin+Riedmiller%2C+Andreas+K%2E+Fidjeland%2C+Georg+Ostrovski%2C+Stig+Petersen%2C+Charles+Beattie%2C+)
21. Rémi Munos, Thomas Stepleton, Anna Harutyunyan, and Marc G. Bellemare. Safe and efficient off-policy reinforcement learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, pp. 1054–1062, USA, 2016. Curran Associates Inc. ISBN 978-1-5108-3881-9… [find](https://scholar.google.com/scholar?q=Re%CC%81mi+Munos%2C+Thomas+Stepleton%2C+Anna+Harutyunyan%2C+and+Marc+G%2E+Bellemare%2E+Safe+and+efficient+off%2Dpolicy+reinforcement+learning%2E+In+Proceedings+of+the+30th+International+Conference+on+Neural+Information+)
22. Ofir Nachum, Mohammad Norouzi, George Tucker, and Dale Schuurmans. Learning gaussian policies from smoothed action value functions, 2018. URL [https://openreview.net/forum?id=B1nLkl-0Z](https://openreview.net/forum?id=B1nLkl-0Z). [↗](https://openreview.net/forum?id=B1nLkl-0Z)
23. Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML’10, pp. 807–814, USA, 2010. Omnipress. ISBN 978-1-60558-907-7. URL [http://dl.acm.org/citation.cfm…](http://dl.acm.org/citation.cfm…) [↗](http://dl.acm.org/citation.cfm…)
24. Gerhard Neumann. Variational inference for policy search in changing situations. In Proceedings of the 28th International Conference on International Conference on Machine Learning, ICML’11, pp. 817–824, USA, 2011. Omnipress. ISBN 978-1-4503-0619-5. URL [http://dl.acm](http://dl.acm). org/citation.cfm?id=3104482.310… [↗](http://dl.acm)
25. Gerhard Neumann and Jan R. Peters. Fitted q-iteration by advantage weighted regression. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou (eds.), Advances in Neural Information Processing Systems 21, pp. 1177–1184. Curran Associates, Inc., 2009. URL [http://papers.nips.cc/paper/3501-fitted-q-iter…](http://papers.nips.cc/paper/3501-fitted-q-iter…) [↗](http://papers.nips.cc/paper/3501-fitted-q-iter…)
26. Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, July 2017. [find](https://scholar.google.com/scholar?q=Deepak+Pathak%2C+Pulkit+Agrawal%2C+Alexei+A%2E+Efros%2C+and+Trevor+Darrell%2E+Curiosity%2Ddriven+exploration+by+self%2Dsupervised+prediction%2E+In+The+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition+%28CVPR%29)
27. Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: Example-guided deep reinforcement learning of physics-based character skills. ACM Trans. Graph., 37 (4):143:1–143:14, July 2018. ISSN 0730-0301. doi: 10.1145/3197517.3201311. URL [http://doi.acm.org/10.1145/3197517.3201…](http://doi.acm.org/10.1145/3197517.3201…) — [doi:10.1145/3197517.3201311](https://doi.org/10.1145/3197517.3201311) [↗](http://doi.acm.org/10.1145/3197517.3201…)
28. Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th International Conference on Machine Learning, ICML ’07, pp. 745–750, New York, NY, USA, 2007. ACM. ISBN 978-1-59593-793-3. doi: 10.1145/1273496.1273590. URL ht… — [doi:10.1145/1273496.1273590](https://doi.org/10.1145/1273496.1273590) [find](https://scholar.google.com/scholar?q=Jan+Peters+and+Stefan+Schaal%2E+Reinforcement+learning+by+reward%2Dweighted+regression+for+operational+space+control%2E+In+Proceedings+of+the+24th+International+Conference+on+Machine+Learning%2C+ICML+%E2%80%9907%2C+pp%2E)
29. Jan Peters, Katharina Mülling, and Yasemin Altün. Relative entropy policy search. In Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence, AAAI’10, pp. 1607–1612. AAAI Press, 2010. URL [http://dl.acm.org/citation.cfm?id=2898607.2898863](http://dl.acm.org/citation.cfm?id=2898607.2898863). [↗](http://dl.acm.org/citation.cfm?id=2898607.2898863)
30. Vitchyr Pong. Rlkit. [https://github.com/vitchyr/rlkit](https://github.com/vitchyr/rlkit), 2019. [↗](https://github.com/vitchyr/rlkit)
31. Doina Precup, Richard S. Sutton, and Sanjoy Dasgupta. Off-policy temporal difference learning with function approximation. In Proceedings of the Eighteenth International Conference on Machine Learning, ICML ’01, pp. 417–424, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1-55860-… [find](https://scholar.google.com/scholar?q=Doina+Precup%2C+Richard+S%2E+Sutton%2C+and+Sanjoy+Dasgupta%2E+Off%2Dpolicy+temporal+difference+learning+with+function+approximation%2E+In+Proceedings+of+the+Eighteenth+International+Conference+on+Machine+Learning)
32. Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations. In Proceedings of Robotics: Science and Systems (RSS), 2018. [find](https://scholar.google.com/scholar?q=Aravind+Rajeswaran%2C+Vikash+Kumar%2C+Abhishek+Gupta%2C+Giulia+Vezzani%2C+John+Schulman%2C+Emanuel+Todorov%2C+and+Sergey+Levine%2E+Learning+Complex+Dexterous+Manipulation+with+Deep+Reinforcement+Learning+and+Demons)
33. John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1889–1897, Lille,… [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Sergey+Levine%2C+Pieter+Abbeel%2C+Michael+Jordan%2C+and+Philipp+Moritz%2E+Trust+region+policy+optimization%2E+In+Francis+Bach+and+David+Blei+%28eds%2E%29%2C+Proceedings+of+the+32nd+International+Conferen)
34. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017. URL http://arxiv.org/abs/1707.06347. [↗](http://arxiv.org/abs/1707.06347)
35. SFU. Sfu motion capture database. [http://mocap.cs.sfu.ca/](http://mocap.cs.sfu.ca/). Richard S. Sutton and Andrew G. Barto. Introduction to Reinforcement Learning. MIT Press, Cambridge, MA, USA, 1st edition, 1998. ISBN 0262193981. [↗](http://mocap.cs.sfu.ca/)
36. Richard S Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In S. A. Solla, T. K. Leen, and K. Müller (eds.), Advances in Neural Information Processing Systems 12, pp. 1057–1063. MIT Press, 2000. [find](https://scholar.google.com/scholar?q=Richard+S+Sutton%2C+David+A%2E+McAllester%2C+Satinder+P%2E+Singh%2C+and+Yishay+Mansour%2E+Policy+gradient+methods+for+reinforcement+learning+with+function+approximation%2E+In+S%2E+A%2E+Solla%2C+T%2E+K%2E+Leen%2C+and+K%2E+Mu%CC%88ller)
37. Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Rémi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample efficient actor-critic with experience replay. CoRR, abs/1611.01224, 2016. URL http://arxiv.org/abs/1611.01224. [↗](http://arxiv.org/abs/1611.01224)
38. Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn., 8(3-4):229–256, May 1992. ISSN 0885-6125. doi: 10.1007/BF00992696. URL https://doi.org/10.1007/BF00992696. [↗](https://doi.org/10.1007/BF00992696)
39. He Zhang, Sebastian Starke, Taku Komura, and Jun Saito. Mode-adaptive neural networks for quadruped motion control. ACM Trans. Graph., 37(4):145:1–145:11, July 2018. ISSN 0730-0301. doi: 10.1145/3197517.3201366. URL [http://doi.acm.org/10.1145/3197517.3201366](http://doi.acm.org/10.1145/3197517.3201366). — [doi:10.1145/3197517.3201366](https://doi.org/10.1145/3197517.3201366) [↗](http://doi.acm.org/10.1145/3197517.3201366)

*Updated 2 November 2026*