CDC 2019 Tutorial - From Self-Tuning Regulators to Reinforcement Learning and Back Again

Abstract

Machine and reinforcement learning (RL) are increasingly being applied to plan and control the behavior of autonomous systems interacting with the physical world. Examples include self-driving vehicles, distributed sensor networks, and agile robots. However, when machine learning is to be applied in these new settings, the algorithms had better come with the same type of reliability, robustness, and safety bounds that are hallmarks of control theory, or failures could be catastrophic. Thus, as learning algorithms are increasingly and more aggressively deployed in safety critical settings, it is imperative that control theorists join the conversation. The goal of this tutorial paper is to provide a starting point for control theorists wishing to work on learning related problems, by covering recent advances bridging learning and control theory, and by placing these results within an appropriate historical context of system identification and adaptive control.

Slides

• Introduction by Anders Rantzer.

• From Self-Tuning Regulators to Reinforcement Learning and Back Again by Nikolai Matni.

• Optimal Adaptive Control in Discrete Systems by Alexandre Proutiere.

• Dual Control: Optimization Based Exploration/Exploitation by Anders Rantzer.

• Concentration Inequalities for System Identification by Stephen Tu.

Papers

Tutorial Papers

• N. Matni, A. Proutiere, A. Rantzer and S. Tu, From Self-Tuning Regulators to Reinforcement Learning and Back Again, IEEE Conference on Decision and Control, 2019.
  

• N. Matni, and S. Tu, A Tutorial on Concentration Bounds for System Identification IEEE Conference on Decision and Control, 2019.
  

Additional References

From Self-Tuning Regulators to Reinforcement Learning and Back Again:

• S. Dean, H. Mania, N. Matni, B. Recht, and S. Tu, On the sample complexity of the linear quadratic regulator, Journal of Foundations of Computational Math, 2019.

• S. Dean, H. Mania, N. Matni, B. Recht, and S. Tu, Regret bounds for robust adaptive control of the linear quadratic regulator, NeurIPS, 2018.

• S. Dean, S. Tu, N. Matni, B. Recht, Safely learning to control the constrained linear quadratic regulator, IEEE American Control Conference, 2019.

• Y.-S. Wang, N. Matni, and J. C. Doyle, A system level approach to controller synthesis, IEEE Transactions on Automatic Control, 2019.

• H. Mania, S. Tu, B. Recht, Certainty Equivalence is Efficient for Linear Quadratic Control, NeurIPS, 2019

Optimal Adaptive Control in Discrete Systems:

• J. Ok, A. Proutiere, D. Tranos, Exploration in structured RL, NeurIPS, 2018.

• A. N. Burnetas, M. N. Kathehakis, Optimal Adaptive Policies for Markov Decision Processes, Mathematics of Operations Research, 1997.

• T. Jaksch, R. Ortner, P. Auer, Near-optimal Regret Bounds for Reinforcement Learning, NeurIPS, 2009.

• S. Agrawal, R. Jia, Optimistic posterior sampling for reinforcement learning: worst-case regret bounds, NeurIPS, 2017.

• Y. Jedra, A. Proutiere, Sample complexity lower bounds for linear system identification, IEEE Conference on Decision and Control, 2019.

• Y. Abbasi-Yadkori et al., Online Learning in Markov Decision Processes with Adversarially Chosen Transition Probability Distributions, NeurIPS, 2013.

Dual Control: Optimization Based Exploration/Exploitation:

• A. Rantzer, Minimax Adaptive Control for State Matrix with Unknown Sign, arXiv:1912.03550, 2019.

Concentration Inequalities for System Identification:

• M. Simchowitz et al., Learning Without Mixing: Towards a Sharp Analysis of Linear System Identification, Conference on Learning Theory, 2019.

• T. Sarkar, A. Rakhlin, Near optimal finite time identification of arbitrary linear dynamical systems, ICML 2019.

• A. Rantzer, Concentration Bounds for Single Parameter Adaptive Control, IEEE American Control Conference, 2018.

• Y. Abbasi-Yadkori et al., Online Least Squares Estimation with Self-Normalized Processes: An Application to Bandit Problems, Conference on Learning Theory, 2011.

• B. Bercu, A. Touati, Exponential inequalities for self-normalized martingales with applications. Annals of Applied Probabability, 2008.

• H.B. Mann, A. Wald, On the Statistical Treatment of Linear Stochastic Difference Equations, Econometrica, 1943.

• J.S. White, The Limiting Distribution of the Serial Correlation Coefficient in the Explosive Case, Annals of Mathematical Statistics, 1958.