I can hardly believe it — the 3rd edition of my book has just come off the press! It feels like only yesterday, back in 2017, when I was sitting on a balcony in Montenegro, contemplating an email from my publisher. The message essentially said, “Hey, would you consider writing a book about Deep Reinforcement Learning?” It was a daunting decision, but curiosity got the best of me. “Why not give it a shot? I have nothing to lose,” I thought.
Seven years later, I’m thrilled to share that the 3rd edition was published just a week ago — what an incredible journey it has been!
Why This Book?
Let me give you an overview of the book and why you might find it worth reading. The book explores the fascinating field of Deep Reinforcement Learning (DRL), a subfield of Machine Learning focused on training agents to make decisions and improve their behavior through interactions with complex environments.
Opinions about Reinforcement Learning (RL) and Deep Reinforcement Learning vary widely. Some see it as “the most important step toward achieving AGI (Artificial General Intelligence),” while others dismiss it, claiming that “random search outperforms RL and makes it redundant.” Such polarized views are perfectly normal — they signal that the field is both dynamic and significant. Indifference, on the other hand, would be a far worse sign for any area of research.
When I first started writing about RL, my main motivation was to bridge the gap between theory and practice. Back in 2018, there was a noticeable lack of quality materials on the topic — not only explaining the theory of RL and Deep RL but also showing how to apply that theory in practice. Judging by the warm reception of the first and second editions, it seems the book hit the mark.
In the third edition, I aimed to preserve the spirit of the book. While covering all major topics in Deep RL, I’ve kept the theoretical explanations concise and focused on practical applications wherever possible. Here’s an overview of the key topics covered:
- Value-based methods: Bellman equation, Deep Q-Networks (DQN), and their extensions and modifications
- Policy-based methods: Cross-Entropy Method, REINFORCE, Advantage Actor-Critic (A2C), and its asynchronous variant (A3C)
- Continuous control problems and their specific methods: DDPG, D4PG, ACKTR, and Soft Actor-Critic
- Trust region methods to improve training stability: PPO and TRPO
- Non-gradient methods (also known as black-box methods): Genetic Algorithms and Evolution Strategies
- Exploration challenges in RL, including advanced exploration methods
- Reinforcement Learning with Human Feedback (RLHF): A method gaining popularity in LLM pipelines but originally developed for RL
- Monte Carlo Tree Search-based methods: AlphaGo Zero and MuZero
- Discrete optimization problems using RL
In addition to these methods, the book features several chapters tackling complex real-world problems using RL approaches, such as:
- Automation of web navigation
- Stock trading
- Interactive fiction games
From my perspective, these examples help readers look beyond traditional problems and environments, preparing them to apply RL techniques “in the wild.”
What’s New in the Third Edition?
If you’ve read the second edition, you’ll notice that there aren’t many major changes. However, here’s what’s new:
- A new chapter on Reinforcement Learning with Human Feedback (RLHF)
- The chapter on AlphaGo Zero has been expanded to include the MuZero method
- Significant rewrites of some chapters, such as the browser automation chapter, which now uses the MiniWoB++ library from the Farama Foundation (MiniWoB++). This update reduces the technological overhead needed to get the environment running and makes the experimentation much simpler.
The book continues to use PyTorch as the core deep learning library, but all examples have been updated to the latest version (PyTorch 2.5).
To fit within the page limit (800 pages is still quite a lot), some chapters from the second edition were removed, including:
- The chapter on RL in robotics
- The chapter on chatbots
- The chapter on imagination-based methods
As before, all the code is freely available in the GitHub repository, and you’re welcome to explore and modify it: GitHub — Deep Reinforcement Learning Hands-On (3rd Edition).
Get the Book
You can find the book on Amazon here: Deep Reinforcement Learning Hands-On (3rd Edition).
Thanks for reading! I hope you’ll enjoy the book as much as I’ve enjoyed writing it.