Reinforcement Learning: An Introduction (Adaptive Computation and Machine Learning Series) [Hardcover]

Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In this book, we provide an explanation of the key ideas and algorithms of reinforcement learning. The discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. This book is meant to be an introductory treatment of reinforcement learning, emphasizing foundations and ideas rather than the latest developments and mathematical proofs. We divide the ideas underlying the field into a half dozen primary dimensions, consider each in detail, and then combine them to form a much larger space of possible methods including all the most popular ones from Q-learning to value iteration and heuristic search. In this way we have tried to make the book interesting to both newcomers and experts alike. We have tried to make the work accessible to the broadest possible audiences in artificial intelligence, control engineering, operations research, psychology, and neuroscience. If you are a teacher, we urge you to consider creating or altering a course to use the book. We have found that the book works very well as the text for a course on reinforcement learning at the graduate or advanced undergraduate level. The eleven chapters can be covered one per week. Exercises are provided in each chapter to help the students think on their own about the material. Answers to the exercises are available to instructors, for now from me, and probably later from MIT Press in an instructor's manual. Programming projects are also suggested throughout the book. Of course, the book can also be used to help teach reinforcement learning as it is most commonly done now, that is, as part of a broader course on machine learning, artificial intelligence, neural networks, or advanced control. I have taught all the material in the book in as little as four weeks, and of course subsets can be covered in less time. Table of contents: Part I: The Problem 1 Introduction 2 Evaluative Feedback 3 The Reinforcement Learning Problem Part II: Elementary Methods 4 Dynamic Programming 5 Monte Carlo Methods 6 Temporal Difference Learning Part III: A Unified View 7 Eligibility Traces 8 Generalization and Function Approximation 9 Planning and Learning 10 Dimensions of Rreinforcement Learning 11 Case Studies For further information, see http://envy.cs.umass.edu/~rich/book/the-book.html.