In the scene of document verification， the standard deep learning face recognition method has low accuracy and poor real-time performance on embedded devices. To solve these problems， this paper proposes a modified efficient convolutional neural network （CNN） called Lightnet and adopts the transfer learning method. Lightnet is an efficient CNN module composed of depthwise separable convolution， linear bottleneck structure and attention module. After introducing the loss function AM-Softmax with additive angle margin， the network model can effectively solve the problems of redundancy parameter and vast calculation for standard CNN in the foundation of ensuring the high accuracy of face recognition. The transfer learning method can enhance the scene-identity face matching performance by freezing all the convolution layer weights of the pre-trained model and fine-tuning training in the self-made scene-identity face matching dataset. The experimental results show that the designed efficient scene-identity face matching algorithm has achieved good results in terms of verification accuracy， parameters and verification speed， and has good robustness in life scenarios.