Online Keyframe Selection Scheme for Semantic Video Segmentation

In this paper, we propose an online keyframe selection scheme for video analysis tasks based on deep reinforcement learning. Optimal keyframe selection is a crucial job for preserving the temporal updates in the video sequence since it affects the accuracy and throughput of the whole video analysis framework to a great degree. Our scheme is generic and can be used for the simultaneous decision of keyframe in any real-time video based task. We employ policy gradient reinforcement strategy to learn policy function for maximizing the expected reward. The proposed selection network has two actions (key and non-key) in the action space. State information is derived from the element-wise difference image of keyframe and the current frame. We adopt deep feature flow approach for feature propagation between keyframe and the current frame. We evaluate our scheme on the Cityscapes dataset with DeepLabv3 as segmentation network and LiteFlowNet for computing flow fields.