Deep reinforcement learning has achieved great success in many applications. However， it usually needs large amount of interactions with the environment to learn the policy， which leads to inefficient training. Imitation learning is an important approach to tackle this challenge by learning from demonstrations， but it instead requires a large set of demonstrations provided by experts， which could be rather costly in many complex tasks. In this paper， we propose an active learning method to reduce the demonstration cost by actively selecting starting state for demonstration. The method is based on uncertainty sampling and dissimilarity sampling. It selects the best state from the candidate set and then queries expert for fixed length of trajectory， in order to train effective policy with fewer demonstrations. Experimental results in multiple environments demonstrate that the proposed method can achieve effective performance with significant lower demonstration cost.