The application of fibrous porous material (FPM) in the field of thermal insulation materials is widely concerned, and vacuum process is one of the most effective methods to improve the thermal insulation performance. The effect of microstructure of FPM on its effective thermal conductivity under vacuum is studied. The existing random structure generation method is improved and D3Q19 lattice-Boltzmann method is used to solve the thermal conductivity. The modified generation method can significantly optimize fiber distribution and reduce the interpenetration rate to 3.1%. The simulation results are in good agreement with experimental and theoretical data. Results indicate that FPM with finer diameter leads to smaller pore size and has a more excellent ability to maintain the lower effective thermal conductivity under higher pressure in the range of 1—8 μm. Besides, the more the length direction of fiber is inconsistent with the heat transfer direction, the better the insulation performance is. And the insulation performance reaches the peak when the direction angle reaches 90°. These results are of great significance to the structural design and optimization of the fiber core material of vacuum insulation panel.