Due to factors such as illumination changes， camera jitter and dynamic background， existing foreground subtraction algorithms cannot achieve good segmentation results in complex scenes. To solve this kind of problems， this paper proposes a subtraction algorithm based on a fully convolutional siamese neural network， which can accurately segment the foreground with only two arbitrary frames. Specifically， the input two images are divided into the base image and the image to be segmented. The algorithm uses the fully-convolutional siamese network to get the similarity metric map of input frames. The similarity metric map contains information about changes in pixels of the image to be segmented relative to the base image. Then， the similarity metric map is fused with the image to be segmented， and the encoder-decoder network is used to achieve end-to-end foreground subtraction results. The paper evaluates the proposed algorithm on the CDnet2014 dataset to prove its effectiveness.