DEEP REINFORCEMENT LEARNING IN PHYSICS-BASED SIMULATIONS

Abstract

In neuroscience, reinforcement learning is an important concept for the learning process of all organisms. Tunicata, a marine invertebrate animal, has during larval stage a primitive brain and eyes, swims around and learns to find the best rock to attach itself into. In the adult stage it digests its brain, emphasizing that the point of having a brain is to make decisions and take intelligent actions. In computer science, reinforcement learning (RL) is a mathematical framework based on Markov Decision Processes, concerned with building rational agents that act so as to achieve the best expected outcome, whilst interacting with an environment without an explicit teacher. Deep reinforcement learning (Deep RL) augments the foundational work in RL with neural networks to solve more complicated tasks, like games, physics-based simulations and robotics. In robotics, physics-based simulations are crucial for training real-life robots. Simulations have seen adoption accelerated by the rapid growth in computational power over the last three decades [1]. Robots are very complicated systems, training them in the real world can be challenging, since execution and feedback is slow. Physics-based simulation allows sampling experience millions times faster than in the real world, making it possible to train very complicated robots. In the first chapter of this thesis, I give a brief introduction on RL theoretical fundamentals. In the second chapter, I introduce the theoretical background behind deep RL methods. In the third chapter, I evaluate the performance of deep RL methods in physics-based simulations with MuJoCo, an excellent engine for advanced physics-based simulations. In the fourth chapter, I research the application of off-policy learning methods in robotics simulations. I evaluate the performance of off-policy learning methods in Fetch mobile manipulator, a 7-DoF robotic arm with a two-fingered parallel gripper. Finally, I draw concluding remarks.