Model-based reinforcement learning approach for planning in self-adaptive software system

Policy-based adaptation is one of interesting topics in selfadaptive software research community. Current works in the field proposed the term of policy evolution, which concentrate to tackle the impact of environmental uncertainty on adaptation decision. These works adopted the advances of Reinforcement Learning (RL) to continuously optimize system behavior in run-time. However, there are several issues remain very primitive in current researches, especially the arbitrary exploitation-exploration trade-off and random exploration, which could lead to slow learning, hence, frail decision in exceptional situations. With model-free approach, these works could not leverage the knowledge about underlying system, which is essential and plentiful in software engineering, to enhance their learning. In this paper, we introduce the advantages of model-based RL. By utilizing engineering knowledge, system maintains a model of interaction with its environment and predicts the consequence of its action, to improve and guarantee system performance. We also discuss the engineering issues and propose a procedure to adopt model-based RL to build a self-adaptive software and bring policy evolution closer to real-world applications.