This book provides a broad introduction to algorithms for decision making under uncertainty. We cover a wide variety of topics related to decision making, introducing the underlying mathematical problem formulations and the algorithms for solving them.


The full book is available as a PDF. You can also download individual chapters. The PDF is shared under a under a Creative Commons CC-BY-NC-ND license.

The copyright of this book has been licensed exclusively to The MIT Press. All inquiries regarding rights should be addressed to The MIT Press, Rights and Permissions Department.

We will be making improvements based on community feedback. It will be released in print in early 2022, but this electronic version will remain publicly available.

To be notified when the print version is available, you can provide your email address here.


  1. Introduction

Part I: Probabilistic Reasoning

  1. Representation
  2. Inference
  3. Parameter Learning
  4. Structure Learning
  5. Simple Decisions

Part II: Sequential Problems

  1. Exact Solution Methods
  2. Approximate Value Functions
  3. Online Planning
  4. Policy Search
  5. Policy Gradient Estimation
  6. Policy Gradient Optimization
  7. Actor-Critic Methods
  8. Policy Validation

Part III: Model Uncertainty

  1. Exploration and Exploitation
  2. Model-Based Methods
  3. Model-Free Methods
  4. Imitation Learning

Part IV: State Uncertainty

  1. Beliefs
  2. Exact Belief State Planning
  3. Offline Belief State Planning
  4. Online Belief State Planning
  5. Controller Abstractions

Part V: Multiagent Systems

  1. Multiagent Reasoning
  2. Sequential Problems
  3. State Uncertainty
  4. Collaborative Agents


  1. A: Mathematical Concepts
  2. B: Probability Distributions
  3. C: Computational Complexity
  4. D: Neural Representations
  5. E: Search Algorithms
  6. F: Problems
  7. G: Julia


We are interested in all forms of feedback including, but not limited to: errors, improvements to code (especially improvements for clarity over speed), typos, areas that are confusing, critical topics that are missing, and ideas for examples or exercises.

Please file issues on GitHub or email the address listed at the bottom of the pages of the PDF.