Hierarchical Imitation Learning with Vector Quantized Models

The ability to plan actions on multiple levels of abstraction enables intelligent agents to solve complex tasks effectively. However, learning the models for both low and high-level planning from demonstrations has proven challenging, especially with higher-dimensional inputs. To address this problem, we propose to use reinforcement learning to identify subgoals in expert trajectories by associating the magnitude of the rewards with the predictability of low-level actions given the state and the chosen subgoal. We build a vector-quantized generative model for the identified subgoals to perform subgoal-level planning. In experiments, the algorithm excels at solving complex, long-horizon decision-making problems outperforming state-of-the-art. Because of its ability to plan, our algorithm can find better trajectories than the ones in the training set.

 Alexander Ilin is a Professor of Practice in the Department of Computer Science of Aalto University. He was a research scientist at Curious AI, Amazon and Analyse2. He obtained his PhD degree from Helsinki University of Technology under the supervision of Prof. Erkki Oja and Dr. Harri Valpola. His research interests focus on deep representation learning and model-based reinforcement learning.